The Study Of Pruning Methods Of Deep Neural Network

Deep learning,a machine learning technique which learns from data by deep neu-ral network(DNN),has become popular in many areas such as image classification,speech recognition and natural language processing.However,due to their massive pa-rameters,current DNN models usually have large storage overhead and low inference speed,which may hinder their application to hardware constrained devices like mobile phones or wearable smart devices.To solve this problem,researchers have proposed many approaches to compress and accelerate DNN,among which pruning is a simple but effective method.Pruning is to remove some unimportant parts while maintaining the model's origi-nal performance.According to the difference of removed parts,pruning can be divided into weight pruning and neuron pruning.The former removes unimportant weights in DNN,while the latter directly removes unimportant neurons.Although DNN pruning is getting more and more attention from researchers,there are still many important is-sues that have not been well studied.We think the following three problems are worth exploring:1)how to perform DNN training and pruning simultaneously.2)how to design a more efficient neuron pruning approach.3)how to apply DNN pruning to specialized tasks.In this dissertation,these three issues will be investigated.The main contributions of this dissertation can be summarized into the following three aspects:1.This dissertation proposes an approach to perform DNN training and pruning si-multaneously.Previous DNN pruning methods relied on a pretrained reference model,thus these methods consist of two stages:DNN training and DNN pruning.However,training a deep model itself is time-consuming and laborious.This dis-sertation first analyzes the importance of pruning threshold to the whole pruning algorithm,then puts forward a dynamic and adaptive threshold framework(DAT).Within DAT framework,the pruning threshold can not only change dynamically according to DNN training process,but also be adaptive to the distribution of weights during training,thus DNN weight pruning can be completed at the end of training.2.Based on the optimization of nonlinear reconstruction error(NRE),this disser-tation proposes a more efficient layer-wise neuron pruning method LNP-NRE.Different from linear reconstruction error which is used in previous layer-wise neuron pruning method,LNP-NRE takes the nonlinear activation function of deep neural network into consideration,so it is a more reasonable optimization objec-tive.Experimental results show that LNP-NRE can prune more neurons than state-of-the-art methods under the same level of accuracy drop.3.To tackle fine-grained image classification tasks,this dissertation proposes an attention-based channel pruning method(ACP)for convolutional neural network(CNN).CNN is a class of deep neural network which is specially designed for processing image related tasks,and CNN channel pruning is corresponding to DNN neuron pruning.The pruned network usually runs on personal devices such as mobile phones or wearable devices,which means that small-scale fine-grained classification tasks are more common in realistic scenarios.However,training a large CNN model directly on these tasks will lead to severe overfitting due to the lack of data.So this dissertation first transfers an existing CNN to fine-grained tasks where the existing CNN model is pretrained on large-scale dataset,then proposes an CNN channel pruning approach by adopting an attention mech-anism.Experimental results verify the effectiveness of the proposed attention mechanism.