Research On Compression Method Of Knowledge Distillation And Its Vehicle Terminal Application

While complex deep learning models typically exhibit high performance,the high computational complexity and storage requirements make it more difficult to deploy them on resource-constrained platforms such as vehicle terminal devices and personal mobile devices.Knowledge distillation,as a special model compression method,is a training method in which the teacher model passes knowledge to the student model,which can improve the accuracy of lightweight student model with less inference latency,weaker computational power and less storage to a level close to that of the teacher model,meeting the deployment and operation requirements of resource-constrained platforms.Current knowledge distillation algorithms generally suffer from large variation in knowledge forms,as well as large room for improvement in the compression ratio and post-distillation performance of the models.To address these issues,this thesis investigates new approaches to knowledge distillation and explores their application to vehicle terminals.The main research work in this thesis is as follows:（1）A more multi-granularity knowledge distillation method with adaptive loss function weights is proposed.To address the problems of analyzing the optimal choice of granularity of current multi-grain size knowledge distillation methods and the need for uniformity in the convergence rate of loss functions at each granularity level,the number of self-analyzers is improved and adaptive weights of loss functions for each branch of the self-analysis module are introduced.Experiments on the image classification open datasets Cifar100 and State Farm,and the self-collected driving behaviour dataset UESTCDriving show that the method can improve the learning ability of the student model and has an accuracy improvement of 0.44%～1.07% compared with the original algorithm when fused with other knowledge distillation methods.（2）A knowledge distillation method with soft label entropy reduction is proposed.To address the problem of confusing system information of soft labels in knowledge distillation,the system information entropy value is used to measure the confusion of information contained in soft labels,and the categories with higher information value are marked as coherent targets to exclude the confusing influence of non-coherent labels on soft labels.The soft labels with reduced system information entropy values are used for knowledge distillation training,which helped to improve the student model’s ability to discriminate non-coherent labels,while retaining the inter-class information contained in the soft labels.Experiments on the UESTCDriving and State Farm datasets show that the method led to a maximum of 2.45% accuracy improvement in the student network.（3）Two vehicle systems equipped with trained models with knowledge distillation are designed and implemented using a server-side approach and an vehicle terminal approach respectively.The server-side implementation is able to continuously update the deep learning model library and provide parameter updates to the deployed models of the vehicle terminal system;The vehicle-terminal-side implementation deploy a lightweight network model trained by knowledge distillation on the Android platform to provide real-time recognition of driving behaviour recognition.The system performance tests of the two implementations show that the knowledge distilled lightweight model can achieve 90.82% accuracy in recognition of driving behaviour with an average running time of 234 ms on a resource-constrained platform,which has good accuracy and real-time performance.