Font Size: a A A

Research On Compression Algorithm Of Convolutional Neural Network

Posted on:2020-01-30Degree:MasterType:Thesis
Country:ChinaCandidate:W J JuFull Text:PDF
GTID:2428330599960084Subject:Measuring and Testing Technology and Instruments
Abstract/Summary:PDF Full Text Request
With the advent of the era of big data,the research on convolutional neural networks has deepened.Convolutional neural networks have achieved excellent results in dealing with many problems such as target detection and target tracking in computer vision.However,as the structure of convolutional neural networks continues to deepen,network parameters become more and more,and the amount of calculation is getting higher and higher,the application of convolutional neural networks in mobile devices or embedded devices with limited computing resources has been greatly restricted.The network compression algorithm is an important means to solve the above problems,therefore the research on network compression algorithms is extremely important.This paper belongs to the research direction of network compression,and the research content includes the following aspects:(1)An improved network compression algorithm based on convolution kernel pruning is proposed,the algorithm divides the network compression process into three steps: filtering,pruning,and fine-tuning.In the filtering process of redundant convolution kernel,the algorithm combines the variance of the L2 norm of the output feature map and the L1 norm of the convolution kernel as the basis for discriminating the importance of the convolution kernel.While considering the characteristics of the convolution kernel weight,it also adds the evaluation index of the output feature map,the experiments showed that the pruned network according to the fusion feature gets higher precision after fine-tuning than the pruned network according to the single criterion.In the pruning method,this paper is the overall pruning of the convolution kernel,does not cause the sparse connection of the network structure,does not require the support of the sparse convolution library.In the pruning strategy,this paper proposes nonlinear segmentation pruning strategy,which divides the network pruning process into multiple times,and each pruning is directed to all layers in the network.The experiment proved that the strategy has higher performance than the one-time pruning,and the compression process is faster than the layer-by-layer pruning.(2)An improved network compression algorithm based on knowledge distillation is proposed,based on the knowledge distillation compression algorithm,the algorithm adds hint training phase to the student network.The improved algorithm divides the training of the student network into two phases.In the first phase,a number of corresponding hint layers and guide layers are introduced in the teacher network and the student network,and the teacher network hint layers is used to guide the training of the corresponding student network guide layers.In the second stage,the joint loss function defined by the knowledge distillation algorithm is used to further train the student network to improve student network performance.The experiment proved that the algorithm can transfer the knowledge learned by the teacher network to the student network to a certain extent,at the same time,after joining the hint training phase,the student network trained by the algorithm can achieve higher precision than the student network trained directly by knowledge distillation.
Keywords/Search Tags:convolutional neural network, network compression, convolution kernel pruning, knowledge distillation
PDF Full Text Request
Related items