A comparison of stochastic global optimization methods: Estimating neural network weights

Scope and method of study. The general objective of this study was to determine the speed and accuracy of alternative global optimization methods in relation to a local algorithm for estimating the weights of neural networks. The specific objective was to determine the relative speed and accuracy of nine alternative stochastic global optimization algorithms in relation to a local optimization algorithm for estimating the weights of neural networks. Comparisons of the algorithms were made by performing multiple estimations from random starting values on six function approximation problems and analyzing the running time and distribution of the final objective function values over the multiple estimations.; Findings and conclusions. The results indicated that no single algorithm dominated all others across the training data sets. More importantly, with respect to the research objectives of this study, the local optimization algorithm was not consistently dominated by any of the stochastic global optimization algorithms. On average, the global algorithms marginally outperformed the local algorithm in obtaining a lower local minimum, however, the global algorithms required more computational resources. Therefore, relative to a global algorithm a local algorithm could perform a greater number of restarts increasing the relative performance of the local algorithm. Thus the results in this study indicate that with respect to the specific algorithms studied, there is little evidence to show that a global algorithm should be used over a more traditional local optimization routine for training neural networks. Further, the results indicated that a large number of local minimums exist for all the neural network training data sets considered in this study. Therefore, neural networks should not be estimated from a single set of starting values whether a global or local optimization method is used.