Structured Pruning Of Convolutional Neural Networks With Enhanced Linear Representation Redundanc

Deep neural network is the most important research branch in the direction of artificial intelligence,which has been paid great attention to by both academics and industry.In the past few decades,with the development of the Internet and electronic devices,people’s ability to collect data has made amazing progress.At the same time,major breakthroughs in computing power and storage technology of hardware have provided the necessary material foundation conditions for deep neural networks and promoted the popularization and prosperity of deep neural networks and even the whole field of artificial intelligence.As people’s research on deep neural network becomes more and more advanced,deep neural network shows a trend of getting deeper and wider,which usually brings us more powerful model representation ability and higher target task accuracy.However,at the same time,the number of parameters and the amount of calculation required for the model increase dramatically,making its deployment requirements more stringent and unable to be popularized in the scenarios with limited hardware size,such as mobile devices and embedded systems.The huge amount of computation also makes the response speed of the model slower and can not adapt to the application scenarios with high requirements for real-time performance.The high hardware requirements also significantly raise the threshold of academic research,industrial production and commercial application,so deep neural networks can only find the soil needed for growth in a few universities with sufficient funds and some top large companies.In order to solve the above problems,model compression technology is developed along with the development of deep neural networks,which is a very important research topic and will be studied for a long time.Structured network pruning excels non-structured pruning methods because they maintain the advantages provided by thriving developed parallel computing techniques.For better structured-pruning performance,in this paper,firstly,we present a data-driven loss function term based on the correlation coefficient between different feature maps in the same layer,named CCM-loss.This loss term can encourage the neural network to spontaneously learn more Linear Representation Redundance within the feature maps produced by different filters during the training process.CCM-loss provides us with another universal transcendental mathematical tool besides L*-norm regularization,which concentrates on generating zeros,to generate more redundance but for the different genres.Furthermore,we design a matching channel selection strategy based on principal components analysis to exploit the maximum potential ability of CCM-loss.In this new strategy,we mainly focus on the consistency and integrality of the information flow in the network.Instead of empirically hard-code the retain ratio for each layer,our channel selection strategy can dynamically adjust each layer`s retain ratio according to the specific circumstance of a per-trained model to push the prune ratio to the limit.Notably,on the Cifar-10 dataset,our method brings 93.63% accuracy for pruned VGG-16 with only 1.40 M parameters and 49.60 M FLOPs,the pruned ratios for parameters and FLOPs are90.6% and 84.2%,respectively.In addition,in the Res Net-56 and Res Net-110 models,experiments prove that the accuracy of the model can be maintained at a very high level even if only one channel is retained in some layers of the model by using the proposed method.