Optimal use of regularization and cross-validation in neural network modeling

Scope and method of study. The objective of this work was to effectively improve the generalization performance of multilayer feedforward neural networks in solving nonlinear regression or function approximation problems. We investigated two key methods: regularization and cross-validated early stopping, and developed some new algorithms to extend regularization and cross-validation techniques. These new methods employed a two-stage training strategy. The samples were divided into the training set and validation set at the first stage, and were combined at the second stage. The SDVR (second derivative of validation error regularization) and VBBR (validation-set-based Bayesian regularization) algorithms were derived to update the regularization parameter by minimizing the validation error and maximizing the validation evidence in standard networks and in Bayesian probabilistic networks, respectively. The RTES (retrained early stopping) and BRES (Bayesian regularization with retrained early stopping) procedures were implemented to refine parameter estimation after early stopping. These new methods were compared with the standard Bayesian regularization and early stopping on extensive simulation experiments and on five real-world problems.; Findings and conclusions. The active use of the validation information was found most promising in adapting regularization. Although the validation error and the validation evidence have different meanings, both of them were more directly related to the generalization performance when networks were trained with the SDVR and VBBR algorithms. The RTES procedure overcomes the limitation of conventional early stopping which uses only part of data for parameter estimation. The BRES approach has the advantages of regularization, passive validation, and combined retraining. In the implementation of the RTES and BRES procedures, the new usages of Moody's GPE (generalized prediction error) criterion were proposed and were demonstrated to be effective in determining the termination point of the second stage training. The results from the simulation experiments and real-world applications showed that the newly developed methods produced networks which generalized better than the standard Bayesian method and early stopping procedure. The results and discussions also shed light on the optimal use of these techniques.