Pruning Neural Networks Based On Stochastic Gradient Sparse Optimization

In recent years,convolutional neural networks have achieved a great success in the field of computer vision,such as images classification,semantic segmentation,objection tracking,objection detection.Huge storage cost and computational overhead are the main constraints which hinder the application of convolutional neural networks in mobile devices.Therefore,network compression gradually becomes a very popular direction.Many researchers propose various effective compression methods,such as low-rank approximation,parameter quantization,network pruning,knowledge distillation.This paper proposes three compression methods for compressing and accelerating convolutional neural network.The main work and contributions are as follows:1.Pruning neural networks via stochastic gradient hard thresholdingNetwork pruning is a direct and effective compression method for convolutional neural network which owing huge parameters.This paper proposes a filter pruning method which update the filters' weights by hybrid stochastic gradient hard thresholding algorithm.During the training process,we use hard thresholding to define the importance of filters which is conducted by computing the weights' L1 norm value of filters.Then the filters with small norm values are set to zero which is called soft pruning.After training,we physically prune the model and obtain a compact and effective model which has the almost the same accuracy as the original network.What's more,the pruned model can accelerate the inference time.While we prune Res Net56 with 63% FLOPs,the pruned model's accuracy only descends 0.94% compared with the original network which is achieving the goal of compression and acceleration.2.Knowlegde distillation based deep networks pruning via stochastic gradient hard thresholdingThe pruned small model can usually meet the limitation of computing resources while we adopt CNN in the mobile application of network compression.On the other hand,pruning small model often brings a sharp drop in accuracy.This paper proposes network pruning method along with knowledge distillation and adopt AHSG-HT pruning in the normal pruning process.In the student network's training and pruning process,the features extracted from the teacher network are provided to guide the student network's features.In the experiments,we prune Res Net and VGG for about 40% FLOPs.It is obvious that adopting knowledge distillation can improve the pruned model's accuracy for about 0.3?4.25%.3.Pruning deep networks via Channel Redundancy with Kullback-Leibler DivergenceIt's very crucial to prune the unimportant channels according to the designed pruning standard in network pruning.This paper proposes a structured pruning method based on Kullback-Leibler divergence which can be used to measure the difference between channels and uses the difference to define the important factor of the channel.For structured pruning,we prune the channels of the pre-trained model with small important factors.After pruning,we fine-tune the pruned model until an effective model with comparable performance is obtained.During the pruning progress,we use Kullback-Leibler divergence to find the redundant channels in Res Net.While we set the pruning rate between 10?30%,the accuracy is usually increased by 0.4?0.6% which is compared with baseline.