Study On The Optimization Of Backpropagation Neural Network Classifier

Artificial neural network (ANN), especially Backpropagation (BP) neural networkhas been an important tool to investigate classification problems such as face recognition,character recognition, license plate recognition (LPR), speech, and signal processing dueto strong learning ability. Many researchers agree that the performance of a neural networkdepends strongly on three factors: feature selection method for the training datasets,optimization algorithm for the neural network and the approach to determine the ANNhidden nodes. It is valuable for the theoretical research and business foreground to find abetter feature selection model, the best optimization algorithm and the best appropriatenumber of ANN hidden nodes for improving the accuracy of the ANN classifier.Because high dimensional feature vectors impose a high computational cost as wellas the risk of ‘overfitting’ on training neural network，it’s necessary to eliminate irrelevantfeatures and select the minimally sized subset of features by feature selection aiming atmaintaining or improving the generalization performance of neural network classifier. Onthe other hand, feature selection methods in terms of the classification error rate of ANNclassifier as the evaluation criterion can yield better performance than other methods, andbetter deal with the datasets with many relevant features.A hybrid feature selection model is proposed to select the most significant featuresfrom all potentially relevant features. The model combines a filter with a wrapper. In thefilter, four variable ranking methods are used to pre-rank the candidate features, and thenan initial GA population is produced based on the degree of significance of the re-rankfeatures. In the wrapper, GA algorithm is utilized to search the feature subsets evaluatedby the classification error rate of neural network classifier，which can help find the mostfeature subset. Tests to some datasets demonstrate that the presented model not only canreduce dimensionality of feature subset, but also can improve the accuracy and efficiencyof classification.To improve the disadvantages of BP algorithms, an optimization algorithm calledGTA that combines the advantages of genetic algorithms (GA) and tabu search (TS) is presented. In order to search a promising initial solutions in favor of locating the bestglobal solution within a wide solution space, the training process is divided into two phase,a promising initial solution is first searched by GA algorithm, and next the best solution isselected by tabu search.Unlike other major algorithms, the TS search starts its procedures from a selectedsolution, not from a random solution within the new population produced by GA in thefirst stage. The best solution that does not belong to the tabu list in a set of candidateneighbor solutions becomes the new current solution. To avoid returning to already visitedareas, tabu list is updated by inserting the best solution to replace of the first enteredsolution. While the tabu condition is reached, TS search is reset with center on the newcurrent solution and the halved radius of search area.Experience has shown that the performance of a neural network depends strongly onthe network architecture, especially the number of hidden nodes, and smaller networks arebetter. However, the error surface of a smaller network is more complicated and containsmore local minima compared with a larger network. On the contrary, bigger networks canachieve the desired accuracy, but a more complicated structure than necessary may causeoverfitting the training data, and cannot achieve good generalization performance. Hence,it is still something of an art to choose the appropriate number of hidden nodesfor a neuralnetwork for a given problem. In order to fast find the best number of hidden nodes in aneural network, and reduce the computational cost, a novel “three points search”algorithm is proposed to estimate the necessary number of hidden layer nodes.Provided consider classification error as heuristic function, the relationship betweenthe number of hidden nodes H and classification error is a parabola curve denoted by E(H).Initially, the interval is determined based on the number of hidden nodes computedaccording to six empirical formulas, and then three points denoted asr0H0espectively H_min,H_mid and H_max selected within the interval are updated based on the error relationshipamong the three points. Finally, the best appropriate number of hidden nodes is searchedwithin the just reduced interval.To dynamically adjust hidden nodes, a hybrid learning model is proposed, whichcombines the advantages of genetic algorithm and reinforcement learning (RL) agent. Where, two agents called respectively NN agent and RL Agent cooperate with each otherto finish the task to adjust automatically networks architecture. NN agent is responsible ofpruning or adding hidden nodes to search the best number of hidden nodes and modifyingweights of ANN. While, RL agent receives the reward of NN agent and adjusts thesignificance of hidden nodes according to an explored optimal policy that can yield themaximum reward based on the former experience and accumulated reward. Unlike AMGAalgorithms, it’s not necessary to compute the significance of the merged hidden nodesaccording to an empirical formula that is too difficulty to find. Experimental resultsobtained by some data sets have demonstrated that the presented algorithms not only canreduce computational cost, but also can find the best global solution and improve theaccuracy of classification.