Research Of Computation Efficient Algorithm For Deep Learning

With the rapid development of related datasets and hardware,deep learning has made significant progress in many fields such as computer vision,voice and natural language processing,especially in computer vision,deep convolution neural networks have achieved state-of-the-art performance in image classification,image detection,semantic segmentation and many other tasks,even exceeding the human eye.The better performance of the neural network often means the deeper,wider network structure,more network parameters and more storage and computational complexity,makes it difficult to deploy on embedded systems with limited hardware resources,which means it often can not be successfully deployed on mobile and embedded devices.Therefore,it is necessary to compress and accelerate the large-scale deep neural network in academic and industrial research.Our thesis is mainly based on convolution neural network.Related works have shown that the convolution layer contributes most of the computational complexity in the convolution neural network.In order to remove the redundancy in traditional convolution layer,this thesis introduced an efficient depthwise separable convolution unit.Based on depthwise separable convolution unit,we proposed an efficient method to compress network constructure.The main contents and results of our work are as follows:1.We analyze and compare the computational complexity between conventional convolution layers and depthwise separable convolution unit.The depthwise separable convolution unit decomposes the conventional convolution process into spatial convolution layer for feature extraction and the pointwise convolution for feature combination,which greatly reduces the number of multiplication and addition operations in the convolution process,while still achieving better feature generation effects.Based on the special convolution structure of depthwise separable convolution unit,we propose a channel pruning method,and illustrate the pruning process in detail,combined with experiments to demonstrate the effectiveness of our method.2.The existing knowledge distillation method can quickly transfer the knowledge in a”teacher” network to the ”student” network,which is widely used in various image classification tasks.Inspired by this method,we introduced two knowledge distillation method to help the pruned model get better accuracy?The one is a layer-wise distillation by reconstructing the weight,the other method is introduing the conventional knowledge distillation method into the finetune process,reducing the optimization space of parameters.Both methods have a positive effect on the performance of the pruned model.The test accuracy of the pruned network model has been obviously improved.3.The traditional APoZ-based channel importance evaluation criterion fails to remove the unimportant convolution channel during the pruning process accurately,resulting in a large loss of model accuracy.Inspired by the entropy-based channel importance evaluation criteria,we propose a channel importance evaluation criterion based on information gain.The output of the following convolution unit guides the pruning of each channel in the current layer.We performed experiments on the CIFAR-10 and ILSVRC-12 image classification datasets,the results of the experiments show that the channel pruning method based on the information gain evaluation criterion is better than the other two methods under various pruning ratios.Furthermore,we use the pruned network as the feature extractor on the PASCAL-VOC image detection dataset.The extraction of the basic network has also achieved resonable results,indicating that the channel pruning method is suitable for most image feature representation tasks.