Regularization Methods For Optimizing Feedforward Neural Network Architecture

In recent years,finding the most suitable architecture of feedforward neural networks(FNNs)have attracted great attention.Some studies have proposed automated methods which can find a small but sufficient network structure without additional retraining and adaptation.Regular-ization terms are often introduced into the learning procedure and have shown to be efficient to improve the generalization performance and decrease the magnitude of the network weights.In particular,Lp regularization is used to penalize the sum of the norm of the weights during training.L1 and L1/2 regularizations are the two most popular Lp regularizations.However,the usual Lp regularization is mainly designed for pruning the redundant weights.In other words,L1 and L1/2 regularizations cannot impose sparsity within the group level.In this dissertation,we address the above concerns.First,we investigate a Group Lasso reg-ularization method by considering only the norm of the total outgoing weights from each hidden layer neurons.As a comparison,the popular Lasso regularization method is introduced into stan-dard error function of the network and deal with each weight separately.The numerical results show that our proposed hidden layer regularization method prunes more number of redundan-t hidden layer neurons consistently than Lasso regularization method for each benchmark data sets.Although Group Lasso regularization can prune the redundant hidden layer nodes,it cannot prune any redundant weights of the surviving hidden layer nodes of the neural networks.Second,we propose a Group L1/2 regularization method(GL1/2)for pruning the hidden layer nodes by considering the outgoing weight vector from each hidden layer node as a group.Its advantage is that it can prune not only the redundant hidden nodes but also the redundant weights of the surviving hidden nodes.However,similar to L1/2 regularization,GL1/2 involves a non-smooth absolute value function,which causes oscillation in the numerical computation and difficulty in the convergence analysis.As a remedy,we propose a smooth Group L1/2 regularization method(SGL1/2)by using a smooth function to approximate the absolute value function.Numerical simulations on a few benchmark data sets are carried out,showing that,compared with Group Lasso(GL2),SGL1/2 can achieve better accuracy,and remove more redundant nodes and re-dundant weights of the surviving hidden nodes.A convergence theorem is also proved for the learning process of SGL1/2.