Research On Model Pruning Technology For Deep Convolutional Neural Networks

In recent years,deep learning technology has achieved rapid development with the support of massive data resources and large-scale high-performance computing equipment.As one of the most successful networks in deep learning technology,deep convolutional neural networks have made remarkable achievements in many research fields.However,the excellent performance of convolutional neural networks is inseparable from its huge computing and memory cost,which brings challenges to the application of convolutional neural networks on resource-constrained devices.To achieve the simplification of deep convolutional neural networks,this thesis researches two model pruning algorithms.The main research contents and contributions are as follows:(1)Model pruning algorithm based on Wasserstein metric.This thesis proposes an evaluation criterion called channel discrepancy,which comprehensively measures the substitutability of channel output features and the channel's feature expression ability.In the algorithm flow,first,the output features of each channel in the convolutional layer will be summarized based on the Wasserstein barycenter,and the summarized features are called the basic response of the channel.Then,the channel discrepancy criterion is used to measure the contribution of each channel's output features,and the channel with the smallest discrepancy is regarded as the more redundant channel,thus needs to be pruned from the network.The channel discrepancy evaluation criterion proposed in the algorithm extends the current channel importance evaluation criteria and provides a new perspective for understanding channel redundancy.A large number of comparative experimental results verify the effectiveness of the algorithm.(2)Model pruning algorithm based on attention mechanism.Inspired by that the attention mechanism can make the network pay more attention to the important features,this thesis introduces the attention branch to the convolution layer of the network to capture the weight of each channel's output feature.Since the dense attention obtained by the attention branch cannot be directly used for pruning,the algorithm further maps the weight vector to a sparse probability distribution,resetting the weight of the channel with lower output feature importance to 0.Finally,the channels in the convolutional layer will be pruned according to their channel sparsity.The algorithm combines the pruning process with network training,and the introduced attention branch can realize the evaluation of the importance of the channel at a low cost.Meanwhile,the experimental results on the benchmark datasets and networks also show the effectiveness of the algorithm.