Research On Knowledge Distillation Algorithm For 3D Object Detection

In the past decades,deep learning has flourished,"siege cities and conquer territories" in a large number of fields,surpassing many traditional methods,and truly making our life more convenient.Its success attributes by improvements in hardware and software,enabling models with sufficient capacity to absorb large-scale data.But at the same time,in specific production and life,due to the needs of scenarios and immediacy,such models with large storage requirements and high computational complexity will become difficult to deploy.For this reason,many model compression methods have been proposed.Knowledge distillation is a typical model compression method,which aims to make the student model achieve better accuracy by adding supervision from the teacher model.3D object detection in autonomous driving is an important topic of deep learning today.Since this part of the algorithm is deployed on a vehicle with relatively low computing power,it is necessary to use model compression methods such as knowledge distillation.3D target detection is more complex than 2D target detection,with more modality data types input,and a detection frame with greater output degrees of freedom.How to find a suitable knowledge distillation paradigm among them has become an important topic.This thesis aims at the problems in the current knowledge distillation and 3D target detection,combined with the latest visual 3D target detection model,conducts research in three aspects of knowledge distillation parameter adjustment,cross-modal knowledge distillation,and cross-scale knowledge distillation,and has achieved the following results:(1)Automatic Gradient Blending Algorithm for Knowledge Distillation.In knowledge distillation training,the loss function is composed of task loss and distillation loss.When changing the teacher-student model combination,different knowledge distillation methods or switching data sets,the scale of distillation loss changes greatly,so it takes a lot of time and computing resources to adjust the weight coefficient before the loss.Based on this,this paper proposes an automatic gradient blending algorithm that can be effectively applied to knowledge distillation.By converting the weight adjustment problem into a gradient blending problem,the modulus length and direction of the total gradient update vector are limited,so that the model training step is stable and the iteration direction is reasonable during training.A large number of experiments were carried out on a number of different knowledge distillation methods and a variety of different teacher-student model combinations.Compared with the traditional manual adjustment method,it can use less than 1/10 of the time to achieve the same or even better accuracy.(2)Cross-modal Knowledge Distillation Algorithm for 3D Object Detection.In the field of 3D target detection,there are mainly two modes of data,one is point cloud information collected by LiDAR,and the other is image information collected by camera sensors.Laser point cloud has better detection performance because it can obtain the structural information of objects,but because of the high cost of laser radar,there are currently more pure vision detection solutions.In this paper,we propose a crossmodal knowledge distillation method for the poor performance of pure vision 3D object detection.The laser point cloud model is used as the teacher model,and the dark knowledge is extracted and refined at the BEV features and then passed to the visual model to improve the perception of the visual model.In this paper,under the nuScenes data set,experiments were conducted with the latest BEVFormer and BEVDepth models,and the experimental results in 3D object detection exceeded the current best knowledge distillation level.In a variety of different network combinations,NDS and mAP improve about 2%.(3)Cross-scale Knowledge Distillation Algorithm for 3D Object Detection.Due to the limitation of computing power and the requirement of low latency,the calculation of target detection on vehicle equipment cannot be too complicated.A common method to reduce the computational complexity is to directly reduce the size of the input image.This method is simple,but the problem is that down-sampling image will bring a certain loss of accuracy.Based on this problem,this paper proposes a cross-scale knowledge distillation algorithm,which uses the high-resolution image input model as the teacher model,and effectively supervises by matching the features of the low-resolution student model in the feature pyramid.We conduct experiments on nuScenes to demonstrate that the cross-scale distillation method can effectively alleviate the performance loss of low-resolution models.Compared with the high-resolution model,the accuracy drop is about half that of the unused distillation method.