A New Research On Escaping From Local Minima In Back-Propagation Algorithm

Back-propagation algorithm is a neural network learning algorithm used widely. But the algorithm is based on the steepest descent method, and the method often suffers from a sub-optimal solution or a local minima problem, which limits its scope of application.Starting with analyzing the reasons of sinking into local minima, the author notices that a saturation region exist for an error function in which the corresponding changes in output layer are unnoticeable or even neglected with those of input layer, causing the output layer's lose of sensitivity to input signals and the severe block of the propagation of information.A modified error function has been proposed which is related to the output of the hidden layers as well as that of the network, thereby increasing the sensitivity to the input network and helping it escape from the local minima and converge to the global minima. Especially the improved various parameters in several special circumstances are discussed and the methods of calculating the optimal parameters given. Finally, a modified XOR is used in the stimulant experiments, and the result contrast with the previous BP algorithm indicates that the new back-propagation algorithm can fast escape from the local minima and converge to the global minima while the previous BP algorithm cannot confirm whether the converged minima is the local one or the global one. Furthermore, the converging speed of improved BP algorithm is much higher than that of the simulated annealing algorithm. All the results of the experiment and contrast are just within the expectation.