Research On Model Pruning Algorithms For Deep Convolutional Neural Networks

Deep learning(e.g.,convolutional neural network)has become the most popular solution for many computer vision tasks.The major weakness of deep neural network is its huge computational cost and storage overhead.Hence,deep model may be hard to be deployed on resource constrained devices such as mobile phones or embedded gadgets.A resource constrained scenario means that a computing task must be accomplished with limited resource supply,such as computing time,storage space and battery power.This small device has very strict constraints on inference speed or model size.Hence,how to accelerate deep model's inference speed and reduce parameter number has become an important research field in the computer vision community.On the other hand,many studies have revealed that the deep model suffers from heavy over-parameterization,which means the model's parameters are redundant.This phenomenon provides a theoretical support for model compression.In this thesis,we argue that filter level pruning is an effective method to accelerate inference speed and reduce memory footprint.We studied several important issues about network pruning,including the following aspects,1.We propose Thi Net,a filter level pruning method based on reconstruction error minimization.While most existing pruning methods are heuristic,we formally establish filter pruning as an optimization problem and proposed a greedy solution to solve this problem.We find that if the reconstruction error of next layer's activation can be minimized,removing the corresponding filters will have negligible impact on model accuracy.To further reduce memory footprint,we propose an algorithm named gcos(which is based on channel shuffle)to address the information blocking issue caused by group convolution.Experiments show that Thi Net outperforms previous heuristic algorithms and achieves good generalization ability.2.We propose Auto Pruner,an end-to-end trainable filter level pruning method.Hand-crafted importance criterion plays an important role in previous three-stage pruning pipeline.In order to solve this problem,we propose an end-to-end trainable neural network layer Auto Pruner.The input of Auto Pruner is the activation tensor of current convolutional layer.After a series of computations,this layer will produce a unique index code.During model fine-tuning,this index code will be gradually binarized.So the whole pruning process is more smooth.After training,all the filters and channels corresponding to zero indexes will be removed and has little influence to model accuracy.Experiments show that Auto Pruner achieves better performance than previous three-stage pruning algorithms.3.We propose CURL,a filter level pruning method for compressing with residual connections and limited data.Pruning with residual connections and limited data is very challenging,and we propose a new pruning scheme CURL.For the residual connection,we argue that pruning both channels inside and outside the residual block can achieve better performance.As for pruning on a small dataset,we propose to use several image transformation techniques to expand the training data and to adopt knowledge distillation method to fine-tune the pruned small model.However,the expanded data may be noisy,and we propose a label refine scheme to update the noisy soft target of knowledge distillation.Experiments show that the proposed CURL algorithm can effectively solve these two issues,which is very important for actual applications.