Research On Explainable Conpression Algorithm Of Deep Neural Network

Deep learning has been the foundation of AI's success over the past few years,but the huge computational complexity and enormous storage requirements make their deployment in real-time applications a tremendous challenge,especially on devices with limited resources.Therefore,how to use compression and acceleration technology to apply the model to the real scene has become a research hotspot.At present,most of the neural network compression methods are relatively weak in interpretation.In this paper,the Shapley value and attention mechanism of the interpretable method are selected as the compression basis,and the research is carried out based on convolutional neural network.The main work contents and achievements are as follows:First,based on the theoretical basis of convolutional neural network model compression,a CNN pruning method that gives strong interpretability to the pruning process is proposed,it can reduce the model size,decrease the run-time memory footprint,lower the number of computing operations,and enhance model interpretability at the same time.It takes a large network as the input model,but during the training process,it will identify and prune the unimportant channels according to the Shapley values,thereby generating a compact model.In this paper,we demonstrate the effectiveness of the proposed method on two commonly used image classification datasets,CIFAR-10 and CIFAR-100,through multiple CNN models,such as VGGNet,Dense Net and Rse Net.For VGGNet,interpretative structural pruning gives a 20× reduction in model size and a 5× reduction in computing operations,and the recognition accuracy is only lost by 2.75%.Secondly,a new knowledge distillation method is proposed to improve the accuracy of the pruned model,namely self-attention knowledge distillation.This method allows a model to learn from itself and to be substantially improved without any additional supervision or labels.Specifically,the attention maps extracted from models trained to a reasonable level will encode rich contextual information.The valuable contextual information can be used as a form of ‘free' supervision for further representation learning through performing top-down and layer-wise attention distillation within the network itself.Self-attention knowledge distillation can be easily integrated into any feedforward convolutional neural network without increasing the reasoning time.VGGNet-19,Dense Net-40 and Res Net-101 with interpretable structural pruning were used to verify the results on CIFAR-10 and CIFAR-100 datasets.By using this method,the pruned model achieves the same accuracy as the original model on the premise that the model size and calculation operand are reduced greatly.The results show that self-attentional knowledge distillation can generally improve attentional map at different levels in different networks.