Research Of The Model Compression Algorithm For Deep Neural Network

Promoted by the development of deep neural network,there raise a new wave of artificial intelligence in academia and industry recently.With powerful ability of extracting features,deep neural network has completely surpassed traditional machine learning algorithms across the fields of image and speech recognition,natural language processing,etc.Such remarkable performance of neural network mainly benefits from its complicated structure and massive training data,accompanied by the large amount of parameters and computation of the models,which restrict their deployment in platform with low resources such as mobile devices.Nevertheless,significant portions of parameters in neural network are redundant,which motivate us to perform model compression and acceleration while maintaining the model performance.Based on this background,this thesis focus on the compression and acceleration algorithms of deep convolutional neural network.The main contributions of this thesis are summarized as follows:Inspired by and developed from compression method based on knowledge transfer,a new knowledge about the instance normalization statistics of feature maps is proposed in this thesis,by analyzing the process of feature extraction in convolution layer.By a visualization experiment,this thesis validate that the statistics contain class-discriminative information of the input,which can be applied in the “teacher-student” training strategy.The statistics of instance normalization can help to further improve the classification accuracy of the student network comparing with the other knowledge definition in existing methods.Under given performance requirement,this allows us to design a more lightweight network with higher performance,resulting higher compression and acceleration ratio equivalently.Like other existing methods,the knowledge introduced above does not consider the correlation among different feature maps.Therefore,this thesis consider the scene that a real teacher teaches students and propose a correlative knowledge based on the covariance matrix of feature maps,in order to make student network learn from teacher network with various type of knowledge jointly.The final experiment results indicate that correlative knowledge of covariance matrix can also achieve better performance than existing methods with faster convergence speed.This thesis also propose a self-select soft channel pruning algorithm combining with knowledge transfer method.Following the soft pruning strategy,we attach an updatable weight mask with sparse constrain to each channel in convolution layer.Under the guidance of teacher network,the network being compressed is able to automatically select unimportant channels based on the global pruning rate and weight mask and finally carries out structured pruning.This thesis validate the proposed method across several network architectures and datasets and experimental results show that the proposed method is able to perform network pruning with lower accuracy degradation,consequently obtains better compression and acceleration result.