Research On Convolutional Neural Network Compression Technology Based On Knowledge Distillation

Convolutional Neural Network(CNN)has made remarkable achievements in the field of image analysis with its strong ability of feature extraction and machine learning.CNN with excellent performance is often supported by huge memory overhead and high-performance computing units(such as GPU).When deployed on terminals with limited computing resources or high real-time requirements,it must be alleviated with the help of lightweight strategies.Aiming at the typical compression task requirements of deploying CNN on drones and other platforms,this thesis focuses on the knowledge distillation strategy to study the feasible method of network lightweight.Firstly,aiming at the problem of information redundancy selection in conventional knowledge distillation technology,this thesis proposes a knowledge distillation method based on the channel attention mechanism.It combines the channel attention mechanism with a variety of pooling methods to aggregate the channel information of the feature map.It strengthens the key channel and suppresses the secondary channel and significantly improves the performance of the student network.Many types of experiments are carried out on the public datasets CIFAR-10 and CIFAR-100 using WRN(Wide Residual Network)to verify the effectiveness of the method.Secondly,aiming at the problem of target recognition network compression from the perspective of drones and other aerial detections,combined with the lightweight network specially deployed to the mobile terminal,a set of task-related datasets(including real images and aerial angle simulated inclined images)are constructed to further expand the knowledge distillation method of the channel attention mechanism.Typical experiments show that the network Mobile Net V3-Large can be compressed from 17 MB to 8.4MB,and the unit reasoning time can be reduced from 6.96 ms to2.71ms;The network Mobile Net V3-Small can be compressed from 7.4MB to 4.1MB,and the unit reasoning time can be reduced from 5.96 ms to 2.68 ms.The compressed network has achieved good accuracy on the datasets under different tilt angles.Finally,aiming at the problem that the number of channels in the network before and after compression is inconsistent and stronger compression cannot be implemented when the network is greatly compressed,this thesis proposes a second-order knowledge distillation method based on the attention mechanism.This method adopts a phased strategy to significantly improve the performance of the compressed network.According to the structural characteristics of Res Ne Xt,a lightweight network Res Ne Xt-Mini is designed.Res Ne Xt-35 is compressed on the public dataset NWPU-RESISC45 by using the second-order knowledge distillation method.The accuracy before compression is 91.37% and the unit reasoning time is 5.77 ms.When Res Ne Xt-35 is compressed into Res Ne Xt-mini20,the compression rate is 93.76%,the accuracy after compression is 91.64%,and the unit reasoning time is 2.78 ms.When Res Ne Xt-35 is compressed into Res Ne Xt-mini14,the compression rate is 94.07%,the accuracy after compression is 89.89%,and the unit reasoning time is 2.09 ms.