Regularization Methods For Sparsification Of Feedforward Neural Networks

Due to the rapid development of artificial intelligence,various algorithms of artificial In-telligence have attracted more and more attentions from both scientific research and industrial applications.Artificial neural networks(ANNs)are the artificial Intelligence algorithm inspired by the function and structure of brains,which inherit the learning and reasoning properties of brains.A neural network can be considered as an adaptive nonlinear dynamic model composed of simple artificial neurons,which can approximate arbitrary linear/nonlinear functional rela-tionships.In a feedforward neural network(FNN),the input data is propagated from input layer to output layer without feedback,and FNN is one of the most widely used neural networks.Many network models for different functions are built by using different techniques to improve the FNNs.For example,regularization method can be applied to enforce sparsity on the weights and the structure of the networks.To extract deep level features,autoencoder(AE)and convo-lutional neural network(CNN)can be used as bricks to build deep networks.This dissertation mainly focuses on several different FNN models and combines regulariza-tion methods to study their sparsity and learning algorithms.Sparsity is an important character in neural network research.It can reasonably limit the size of networks and improve the learning efficiency of networks,the generalization of networks and the interpretability of input data.This dissertation uses regularization methods to punish the weights of FNNs directly or indirectly.After punishing,if some weights and the weight vectors of some nodes become zero,the corre-sponding connections and nodes have no contribution to the calculation of the network output,then they are redundant.Accordingly,it can find some redundant connections and nodes,so as to achieve the sparsification of FNNs.The contents of this dissertation are as follows.1.A regularization method is used to identify and to eliminate the redundant dimensions of the input data and the corresponding input nodes,thus to achieve the sparsification of the input layer of FNNs,while training the network.In the existing references,the regularization method is usually applied to the hidden layer for penalizing the individual weight of hidden layer nodes.In this dissertation,the regularization method is used to the input layer for penalizing the whole weight vectors of input nodes.L1 or L1/2 regularizer is introduced into the input layer weights training process.The regularization drives the weight vectors of the input nodes to zero for finding the redundant dimensions and the redundant input nodes without damaging the approximation or classification performance of networks.2.For further study on the sparsification of the input layer,a smooth function is used to replace the absolute value function in the group L1/2 regularizer and to obtain a smooth group L1/2 regularization method for pruning the redundant nodes in the input layer.It is achieved by intoducing a smooth group L1/2 regularizer with respect to the input nodes into the error func-tion so as to drive some outgoing weight vectors from the redundant input nodes to zero during the learning process.In comparison to L1 regularization and group Lasso,smooth group L1/2 regularization can simultaneously achieve the sparsity in weight level and the sparsity in node level.It also can reduces the oscillation caused by the non-smoothness of group L1/2 regulariza-tion at the origin,such to improve the generalization of networks.The convergence theorem of the proposed training algorithm is proved.3.An L1/2 sparse autoencoder is proposed by using/1/2 regularization method to enforce sparsity on the hidden representation of an autoencoder,in which the network weights are indi-rectly punished for achieving better sparse representation.This representation is able to improve the interpretability of the input data and the generalization of a model.The proposed autoencoder is used as bricks to build deep networks for extracting more general and abstract representation of data and achieving more accuracy classification performance.