Sparse Gradient Learning Algorithms For Feedforward Neural Networks Based On Smoothing L₀ Regularization

Scientists have been exploring the process of the human brain to set up a new discipline-neural networks, which have been widely used in various fields in recent years. Feedforward neural network is an important neural network model, and the BP algorithm, which is derived based on gradient method, is one of the most popular learning algorithms for neural networks. A variety of improved algorithms have been proposed based on the gradient method.By adding a regularization term to the common error function, gradient training method with regularization aims to drives the redundant network weights to zero, thus reducing the network complexity and improving the generalization ability of the network. L0 regularization tends to produce the sparsest solution, corresponding to the most parsimonious network structure. However, due to the NP-hard nature as a combinatory optimization problem, it cannot be directly applied to the gradient method to prune neural networks. To this end, this thesis considers gradient training method with smoothing L0 regularization (GTSLO) for feedforward neural networks, where the L0 regularizer is approximated by a series of smoothing functions. The underlying principle for the sparsity of GTSLO is provided, and the convergence of the algorithm is also theoretically analyzed.The first chapter provides a brief introduction for neural networks, including the develop-ment history, learning processes, gradient training algorithms and Lp regularization method. A batch gradient training method with smoothing L0 regularization term is proposed in Chapter 2, where the sparsity and the convergence of the proposed algorithm is theoretically analyzed. In Chapter 3, an online gradient training method with L0 regularization term is proposed, and the corresponding theoretical convergence results, including both the strong convergence and weak convergence, are obtained. The above two proposed algorithms and the corresponding theoret-ical findings are verified with simulation examples, showing the effectiveness and superiority over other similar regularization methods. This thesis is summarized in Chapter 4.