Research On Knowledge Distillation Algorithm Based On Metric Learning

Deep learning is a breakthrough technology in research fields such as autonomous driving,face recognition,biomedical image processing,and robot vision.Although Neural network models based on deep learning and corresponding training algorithms have significant performance in many large public data sets,they are often limited by expensive hardware devices and cannot be applied to small devices such as mobile phones.Knowledge distillation can make small-scale neural networks have similar effects to large-scale networks by learning teacher networks through student networks.Compared with other model compression algorithms,it is more widely applicable to scenarios.Therefore,this thesis focuses on the application of knowledge distillation algorithm in image classification and face recognition tasks.Knowledge distillation algorithms mainly include algorithms based on the output of the teacher network,designing intermediate modules and feature comparison.Among them,the first two methods have poor applicability for certain high-dimensional data sets,and because of the high dimensionality of the image samples,the algorithm that directly compares the characteristics of a single sample between networks also has a certain degree of model scale.Although the scalability of the knowledge distillation algorithm is relatively high,due to the late start of this direction,the algorithm research ideas are relatively simple.Based on this,in order to expand the applicability and training effect of knowledge distillation in deep learning,this thesis proposes a knowledge distillation algorithm based on metric learning and expands the improved algorithm for measuring multiple samples based on this.The main contents include:(1)This thesis proposes a knowledge distillation algorithm FDM(Feature Distance Metric)based on two-tuple metric relations.The algorithm takes the distance between samples as a measurement index,designs a loss function related to this design and applies it to the process of student network learning from teacher network.In order to solve the problem of distance metrics in high-dimensional space,the FDM algorithm uses a kernel function to obtain higherdimensional distance information when calculating metrics.(2)Due to the limited sample structure relationship information reflected by the distance index,this thesis proposes an improved algorithm T-FDM(Triple-Feature Distance Metric)on the basis of FDM algorithm.Based on the FDM algorithm,the T-FDM algorithm adds the angular relationship between the ternary samples as a measurement index,so that the student network can learn the image feature relationship extracted by the teacher network from a higher dimensional level.When dealing with the problem of boundary samples,the knowledge distillation algorithm is combined with the triplet in metric learning to improve classification effect of the student network itself and solve the problems in the teacher network.(3)This thesis conducts experiments on the proposed algorithm on the two tasks of image classification and face recognition and compares the proposed algorithm with several representative knowledge distillation algorithms in the field.During the experiment,in order to calculate the relative relationship of different types of samples more evenly,a new sampling strategy was proposed instead of random sampling,and its effectiveness was verified in the experiment.This thesis discuss the experimental results of the algorithm from the two directions of experimental performance indicators and sample relationship visualization.After that it analyzes the experimental results to verify the effectiveness of the algorithm in this thesis.