Research On Compression Approach Of Network Of General Object Detection Based On Knowledge Distillation

Many deep neural networks contains a large number of parameters and a huge amount of calculations,which limit their application in mobile devices and real-time systems.Therefore,the training of efficient lightweight neural network has become a prerequisite for the implementation of deep learning in some practical applications.The traditional knowledge distillation mainly studies the deep network in the classification field.Aiming at the lack of research in the general object detection field,a lightweight and high-precision detection network is proposed by the method of knowledge distillation to meet the requirements of computing and storage of mobile devices and real-time systems.A "teacher-student" knowledge distillation algorithm,named EncoderMimick,is designed to transfer the powerful recognition ability of the large and high-recognition teacher network to the light and low-recognition student network.Instead of distilling the feature map in a simple and intuitive way,EncoderMimick's basic idea is to use an auto-encoder to encode the high-dimensional feature map into a low-dimensional space to extract the principal information,and then conduct distillation learning in the low-dimensional space.In addition,a loss function is designed to focus on the points with strong response in the feature map of teacher network and the points with great difference between teacher network and student network,and to ensure the stability of training to avoid explosion.When testing the performance of EncoderMimick algorithm,the effectiveness of EncoderMimick algorithm was illustrated and verified by taking the authoritative COCO data set as the benchmark,the mean mean Average Precision(mmAP)as the evaluation metric,and RetinaNet as the basic network structure.When knowledge distillation is conducted in both feature map and classification branch,the mmAP in COCO object detection task of RetinaNet with ResNet-18 as backbone can increase from 0.318 to 0.338 with the help of EncoderMimick.In additionto the detection field,the algorithm has important reference value in knowledge distillation in deep learning fields such as segmentation,tracking and Natural language processing(NLP).