Key Techniques Of Model Compres-Sion For Deep Convolutional Neural Networks

Recently,deep neural networks has made many breakthroughs in image processing,object detection,natural language processing and many other areas.As the scale and depth of neural network increase continuously,the amount of paramter and computation explode as well.Net-work compression is borned to handle this issue by compressing the prameter and computation.However,current network compression algorithms still have some weaknesses:long compression time,only applicable for single scene,lack of fusion between different algorithms.To address these challenges,we aim to further improve the efficiency and propose to investigate novel com-pression algorithms and fusion framework.We mainly focus on the area of object classification and detection.The main contribution of this work can be summarized as follows1.We propose a novel network filter pruning algorithm based on quantized auxiliary layer.We develop a one stage pruning framework by fusing network training with parameter pruning,which abandon the traditional way of designing a pruning criterion.By inserting a trainable quan-tized auxiliary layer after each convolutional layer we can select fitlers by training and the pruned filters can always recover during the training process.In addition,We fix the vanishing gradi-ent problem in quantized auxiliary layer in two ways:gradient estimator and asymptotic gradient sampler2.We propose a novel knowledge transfer algorithm based on normalized matching knowl-edge,which is designed to handle the matching problem between teacher and student.First,we propose to adopt the significance score of channel and space as knowledge.Based on specula-tion that there is a gap between the knowledge of teacher and that of student,we normalize the knowledge from teacher and student and use the normalized knowledge as supervisory signals to supervise the training of student network.3.We propose a compression framework that fused different kinds of compression algo-rithms,which is is composed of pre-training module,pruning module,knowledge transfer module and quantization module.We mainly focus on one-stage object detection models.We first train the network to obtain the full precision model.Second,we use pruning methods to shrink the parameters of the mode,Then we adopt knowledge transfer to remedy the accuracy loss.Finally,we take advantage of 8 bit quantization technique to further compress the storage of parameter and the amount of computaion.In this way,the model can be compressed to a great extent.For the above methods,we conduct plenty of experiments on various datasets and intelligent scenes.The experiment results validate the effectiveness of our method and framework.