Research On Adaptive Soft Pruning Algorithm Based On Sensitivity Feedback

In recent years,deep learning technology has developed rapidly.Convolutional neural network,as a pillar technology of deep learning,has become one of the most popular scientific research trends.However,with the increasing demand of engineering applications,the over parameterization problem of convolutional neural network leads to the continuous expansion of network scale and high computing consumption,which makes it very difficult to deploy the model to devices with limited memory and computing power.Aiming at the problem of over redundancy of parameters in neural network,Thesis studies the acceleration and compression algorithm of convolutional neural network model.The main research work is as follows:Aiming at the problem of measuring the importance of different filters in the process of network pruning,an adaptive global pruning algorithm based on sensitivity feedback is proposed.The algorithm first considers the measurement of the importance of the filter in the pruning process.The greater the change of the loss function when a filter is deleted,the more sensitive the model performance is to the change of the filter.Based on this idea,the convolution layer of the network is transformed,and a mask layer is added after the convolution layer.Each filter corresponds to a trainable mask parameter.The low-order term of Taylor expansion of the loss function to this parameter is used to measure the sensitivity of each filter.In order to distinguish the important filter from the redundant filter,a regularization factor is introduced into the loss function to increase the sensitivity variance of the filter in the network.When pruning,the global pruning method is used,and different layers will adaptively find the most appropriate pruning rate.Aiming at the problem that traditional hard pruning may seriously damage the network structure and cause the irreversible decline of network effect,a progressive soft pruning algorithm based on threshold attenuation is proposed.Compared with the traditional pruning method,which directly removes the redundant filter from the network in the pruning process,the algorithm in Thesis uses a threshold that gradually decays to0 to suppress the redundant filter,and continuously improves the pruning rate with the progress of iteration until the pruning rate reaches the predetermined target pruning rate.In addition,the regular penalty term is added to the loss function to reduce the activation difficulty of redundant filters,so that the important filters deleted by mistake can be reactivated in the subsequent training process.Finally,the pruning experiments of VGGNet and Rew Net series networks are carried out on the pytorch deep learning framework,and the experimental results are analyzed and summarized to verify the effectiveness of the proposed algorithm.