Deep Convolutional Neural Networks Pruning Research Based On Similarity

Deep neural network has strong feature learning ability,deep learning technology represented by convolutional neural network has achieved the best results in the field of traditional artificial intelligence,such as computer vision,natural language processing,video processing,etc.In academia and industry The world set off a wave of ”deep learning”.Depth of the depth of the network has a wide range of applications and great needs in various fields,such as voice recognition,real-time translation,security,medical and so on.Transplanting AI technology into the portable lifestyle of the general public has become a necessity.Models for portable mobile devices do not necessarily require the best performance,but they place high demands on the experience in terms of product price,computing resources,deployment space and real-time.However,the rapid growth of network size and complexity makes deep neural networks difficult to deploy in mobile terminals with limited computing resources and high real-time requirements.The research on light-weight of heavy networks has become a hot area at present.In this dissertation,we investigate various work related to network compression and deeply study the operation mechanism of each component and network model of convolutional neural network.Propose a convolutional neural network pruning technique based on similarity.In this dissertation,the pruning method based on neuronal similarity is deeply studied in the full-connect layer.The similar neurons in the full-connect layer are cut off and the connections are deleted to simplify the weight matrix,thereby reducing the size and computational complexity of the convolutional neural network model.We applied standard machine vision data sets MNIST and CIFAR-10,as well as the TENSORFLOW official online standard simple convolutional neural network and advanced convolutional neural network structure.We designed several sets of experiments to explore different full-connect layers pruning sequence between the performance of the impact.Experiments found that small-scale network in the full connection layer cut off more than 80% of the unit did not reduce the network performance,there is redundancy in full-connect layer;at the same time this dissertation designed a number of groups of experiments to explore different pruning sequence of the last pruning Performance,the experimental results show that the performance of unit-pruned models in full-connect layers is also slightly increased in some groups experiments,showing that the model size is not as large as possible.Pruning shallow layer then the deep received better performance,eventually subtract nearly60% of the parameters and the model did not decline performance.Finally,this dissertation makes use of separable-convolution to do some useful experiments for acceleration.It is found that separable-convolution can slightly speed up the operation of small networks,but the performance degradation is larger thus separable-convolution is not suitable for small networks.A pruning method based on the similarity of kernel is also proposed for the convolutional layer,and the similar kernel in the convolutional layer is cut off so as to significantly reduce the computational load of the model.Firstly,the method of kernel merge is demonstrated,to explore the role of different components in the convolutional neural network,the sub-convolution kernels and the large convolution kernels are analyzed and verified in detail.The experimental results show that there is redundancy in the convolution layer.When the full-connect layer is not pruned,the kernel value of more than15% can be cut off and the performance of the model is not degraded.The redundancy also makes the network more robust;different pruning sequence can also bring different pruning performance,first cut conv layer cut the entire connection layer(from shallow to deep)pruning way to achieve better pruning effect,whether the amount of shear parameters Small amplitude or small amount of computation,eventually minus nearly 90% of the parameters and nearly 20% of the calculation;large number of convolution kernel experiments show that the pruning parameters and the amount of computation is trade-off Problems,such as a light-weight network that wants to train balanced performance,must explore great weight space,iteratively prune and optimize the network structure.The pruning idea based on similarity can be applied to all convolutional neural networks,regladless of the database the network trained from,which has universality and redundancy in the network.This makes the pruning scheme in our work can be applied to deep convolutional neural networks.Moreover,many groups of experiments explored the different pruning sequences for heuristic pruning work in the future.In this paper,we deducted nearly 90% of the parameters and nearly 20% of the computation on the advanced convolution neural network,The current field of network compression better results,but only to the network pruning wihtout using quantitative or coding to further compress the network.The shortcoming of the work lies in: Due to the too large weight space and the high complexity of the algorithm,this dissertation does not do experiments in large-scale networks and large-scale database.