Research On Deep Network Compression Method Based On Model Gradient Information

Contemporarily,the tremendous success of excessively large Convolutional Neural Networks has been witnessed,which are substantially boosting their superior performances in various fields of computer vision,such as image classification,semantic segmentation and target detection.With the development of Convolutional Neural Networks,the model has a tremendous number of trainable parameters and huge computational costs.Therefore,it is impractical for over-parameterized networks to deploy on resource-constrained platforms,e.g.,embedded sensors,drones,mobile devices,automated robots,etc.Researchers propose network pruning to reduce the model parameters and computational costs,which can compress and accelerate deep neural networks.For solving the above-mentioned problem of Convolutional Neural Networks,two novel network pruning methods which utilize both gradient information and parameter information are proposed.The major contributions of this thesis are as follows:1.For solving the pruning problem of Convolutional Neural Networks,this thesis proposes a structured pruning method,namely,Filter Pruning via Gradient Support Pursuit(FPGraSP),which is inspired by the idea of Gradient Support Pursuit(GraSP)and extend the sparsity-constrained optimization mentioned above to Convolutional Neural Networks.Some previous structured pruning methods only focus on the parameter information of the Convolutional Neural Network,and the filters with smaller weights need to be pruned.However,even if the weights of some filters are small,the variation of parameters will have a great impact on the results of the network.For this reason,FPGraSP exploits parameter information and gradient information,which can effectively prune the redundant filters of Convolutional Neural Networks.Specifically,the filters with the maximum gradient values are selected in the optimizer step,and their indices are merged with the indices of the filters with the largest weights so that a union is achieved.Thereafter,parameters are updated over the above union.Then,filter selection is utilized in a dynamic way to set the filter with the smaller norm to zero.After training procedure,FPGraSP can obtain the compressed model.Experimental results clearly demonstrate the efficiency of FPGraSP.For example,for pruning ResNet-56 on CIFAR-10,FPGraSP without fine-tuning obtains 0.04% accuracy drop,achieving 52.63% FLOPs reduction.In conclusion,FPGraSP can compress deep Convolutional Neural Networks with maintaining its accuracy.2.The above work uses the norm of gradients to measure the gradient information of the Convolutional Neural Network.However,this thesis questions the norm-based pruning criterion,using the cosine similarity of gradients as a criterion for evaluating the importance of the filters.In this regard,a structured pruning based on gradient similarity is proposed.Specifically,for the pre-trained network,the pruning method proposed above calculates the cosine similarity of the gradient for evaluating the importance of the filter.Then,it prunes the filters which are evaluated as unimportant,and finally fine-tunes the pruned network.In addition,for solving the structured pruning problem of Convolutional Neural Networks,this thesis also proposes a hybrid pruning method to combine the structured pruning method based on gradient similarity with the parameter norm-based criterion.The hybrid pruning method use parameter information as well as gradient information,and considers similarity information.As an example,for pruning ResNet-56 on CIFAR-10 dataset,when reducing 52.63% FLOPs,the hybrid pruning method achieves 0.35% accuracy increase.The experimental results verify that the proposed pruning method can improve the performance of the neural network to some degree.