Convergence Analysis Of Learning Algorithms With Smoothing Regularization For Feedforward Neural Networks

Artificial neural networks have been developed at a fast speed in recent several decades due to their strong non-linear mapping capability. Now, neural networks have been successfully used in various fields such as intelligent information processing, pattern recognition, feature extraction, compressed sensing etc. The gradient method is one of the most famous application on training feedforward neural networks (FNNs for short). There are two practical ways to implement the backpropagation (BP for short) algorithms:batch updating approach and online updating approach. But, the major drawback of FNNs is its slow convergence. Many research results have shown that the smaller network that fits the training samples adequately has better generalization.It is well known that the general drawbacks of BP training schemes are their slow con-vergence and weaker generalization, so penalty term or momentum term are introduced into the updating formula to overcome these drawbacks. Most of the literature focus on L2regularization which is used for reducing the magnitude of the network weights. Another main task is how to simply the network structure so as to get better sparse performance and lower cost. The L1/2regularization method has its particular advantages for sparse performance. But the usual L1/2regularization term is not smooth at the origin, which causes difficulty in the convergence analy-sis and, more importantly, oscillation in the numerical computation as observed in our numerical experiments. To overcome these drawbacks, a modified L1/2regularization term is proposed by smoothing the usual one at the origin.The research of this dissertation focuses on developing a novel method for FNNs to get better performance by introducing an L1/2regularization term or a momentum term. Also, we present the convergence analysis. The organization of this dissertation is as follows.1. In Chapter1, some background information about FNNs is reviewed.2. In Chapter2, we propose the deterministic convergence of a three-layer FNNs with smoothing L1/2regularization. The monotonicity of the error function in the training it-eration is proved. Then, some weak and strong convergence results are obtained. Using the smoothing approximation technique, the deficiency of the normal L1/2regularization term can be overcome, which removes the oscillation of the gradient value and the error function. Meanwhile simulation results support the theoretical findings and demonstrate that our algorithms have better performance than the other three algorithms with L1, L2and normal L1/2regularizations respectively.3. In Chapter3, we consider the convergence of online gradient method for FNNs with smoothing L1/2regularization penalty. The boundedness of the weights are needed for the convergence analysis in most of the existing literature. However, these boundedness conditions for non-regularization algorithms may be hard to check, and there is no theory to guarantee such conditions. In this dissertation the boundedness of the weights during the network training are proved. The conditions to guarantee the convergence are relaxed compared with the existing results. The weak and strong convergence results are obtained.4. In Chapter4, we consider the convergence analysis of a batch gradient algorithm with smoothing L1/2regularization and adaptive momentum for FNNs. The weak and strong convergence results are guaranteed under the conditions that the learning rate is a constant and the momentum factor is adaptive.