Research On Pruning And Knowledge Distillation Method Of Deep Neural Network

With the increase in the performance of deep neural network models,the depth of neural network models is getting deeper and deeper,and the redundancy of network models is increasing.What follows is the shortcomings of high storage and high power consumption of the network model.And it restricts the deployment of neural networks in resource-constrained equipment.Therefore,it is urgent to compress and accelerate the deep neural network model.The compression and acceleration of the model refers to reducing the network scale,storage and calculation amount,so that the network has lower hardware requirements than the network before compression.This paper mainly focuses on research on pruning and knowledge distillation method of deep neural network.The main work includes the following aspects:(1)The existing deep neural network compression methods only consider the importance of convolution kernels or feature maps in the entire network,but the similarity relationship between convolution kernels or feature maps is ignored.A similarity pruning method is proposed,which uses matrix similarity to measure the similarity of convolution kernels or feature maps,and then effectively compresses the structure of network neural networks.In addition,this method does not need to use the fixed global pruning rate,and set the dynamic pruning rate by measuring the degree of dispersion of each layer,thereby realizing dynamic pruning.Finally,we summarized how to select the appropriate pruning parts for pruning operations in neural networks of different depths,which guides our subsequent research and learning.(2)An important pruning algorithm based on similarity is proposed,and the information of convolution and feature map is fully used to study network pruning to achieve model compression(parameter reduction)and acceleration(FLOPs reduction).The previous deep neural network compression methods independently split the convolution kernel or feature map to measure the convolution kernel or feature map retained.Based on the similarity of the convolution kernel,this paper sorts the importance of the feature maps according to the high feature values of the feature maps.The method proposed in this paper can effectively identify and pruning the redundant information in the network in the classification task model.(3)An online knowledge distillation method is constructed to improve the accuracy of the network by knowledge distillation.The same-layer and cross-layer information transfer between the teacher network and the student network to realize online knowledge distillation.In addition,to make better use of the information of the two networks,we fused the output of the two networks to guide the student network.The method proposed in this paper makes full use of the result information and intermediate information of the network to achieve the purpose of knowledge distillation.Finally,in order to prove the effectiveness of the pruning and knowledge distillation method of deep neural network proposed in this paper,it is verified on several classic classification data sets.