Structured Deep Neural Network Compression Based On Computer Vision

In recent years,Deep Convolutional Neural Networks(CNN)have achieved remarkable success in computer vision tasks.However,CNN usually generates a lot of computing and storage consumption,which hinders its deployment on mobile and embedded devices.In order to solve this problem,a lot of research has focused on compressing the scale of CNN.Model compression as an emerging research field can reduce the storage size of network models and speed up their running speed by reducing the redundancy of deep neural networks.It is widely used in computer vision tasks,such as object recognition,classification,and detection.In this thesis,the author will conduct certain research and application exploration on the compression of deep neural network models,based on two common compression methods,parameter pruning,and knowledge distillation.The main work and contributions of this thesis are as follows:1.As a simple and effective model compression method,parameter pruning prunes the redundant weights and retains important weights in the network according to certain standards,which can greatly reduce the number of parameters and compress the space,so as to achieve the purpose of reducing the amount of calculation and storage of the network model.Structured pruning is widely applied in the acceleration of CNN because it avoids the problem of random connection caused by unstructured pruning and leads to irregular storage access.In general,structured pruning is mainly divided into two methods: structured row sparseness and column sparseness.Most of the work is pruning the convolution kernel channel or the convolution kernel itself,which belongs to the row demension pruning.In this thesis,the author based on the theoretical basis that under the same sparsity requirements,selecting weights that require sparseness in the column dimension has more choice than that in the row dimension,and proposes structured column pruning will achieve better performance under the same speedup requirement of the same network.Experimental results show that when GFLOPs are reduced by the same multiple,the accuracy and actual acceleration after column pruning are significantly improved,compared to row pruning based on the two structured pruning.2.This thesis is deeply involved in the formulation of information technology neural network representation and compression standards under the Ministry of Information Industry.According to the requirements of the "A new generation of Artificial Intelligence industry Technology Innovation Strategic Alliance standards(AITISA)Working Group Neural Network Representation and Model Compression Standard Technical Requirements",in terms of the structured sparsity of the model,the author proposes and published a technical proposal "Structured deep neural network compression based on incremental regularization pruning",which solves the problem that the traditional direct assignment of constant regularization penalties leads to the weak expressiveness of the compression model,and promotes the standardization of neural network representation and compression technology.3.Besides parameter pruning,knowledge distillation is another effective neural network compression method.It transfers knowledge from a high-capacity teacher network to a low-capacity student network,so that the student network can obtain better performance.Based on this,this thesis proposes a new method by training multiple student model instances at the same time.This method adds losses of similarity and diversity to the baseline knowledge distillation,and adaptively adjusts the proportion of these losses according to the changes in the accuracy of multiple student examples,so as to build a system that allows students to collaborate and compete with each other to better improve the robustness of the system and the performance of each student network.The performance of this method on various scale data sets is better than the existing offline and online distillation schemes.