Cross-domain Knowledge Distillation Based On Dataset Similarity Measurement

With the continuous development of deep learning,current models in the field of image recognition have a great demand for space resources,computing resources and data resources.However,for some scenes lacking resources,such as automatic driving,how to obtain models with small scale and high precision becomes more important.Among them,for the problem of lack of data resources,cross-domain processing can be adopted,and the source dataset which is close to the target dataset can be selected to expand the data resources.For the problem of lack of space resources and computing resources,the knowledge distillation training model can be adopted.Cross-domain knowledge distillation is formed by combining cross-domain processing with knowledge distillation,so as to solve the shortage of space resources,computing resources and data resources at the same time.This thesis launched research to the cross domain knowledge distillation,the main work is as follows:1.The algorithm of Optimal Transport Dataset Distance(OTDD)was improved.In this thesis,Principal component analysis algorithm is proposed to reduce the dimension of image data and extract the more important features in the data.In this thesis,the label of the dataset is taken as the unit,the data in the label is modeled by Gaussian mixture model,and the distance between the label of the dataset is obtained by calculating the Wasserstein-type distance between the Gaussian mixture model.It is found in this thesis that the accuracy of the proposed algorithm is close to that of the OTDD algorithm on MNIST,Fashion MNIST,KMNIST,EMNIST and USPS datasets,but the time consumed by the proposed algorithm is much less than that of OTDD.When calculating the distance between KMNIST and EMNIST,When the number of Gauss functions in Gaussian mixture model is 3 and the retention dimension of Principal component analysis is 1%,the speed of the proposed algorithm is 4.63 times that of OTDD.2.A cross-domain knowledge distillation algorithm based on block migration is proposed.In this thesis,the distillation process is divided into two parts.In the first stage,the teacher model and the student model are divided into blocks,and the student block is transplanted into the teacher model successively to get several sub-models,and the pre-training is carried out through the training sub-models.In the second stage,knowledge distillation based on attention mechanism is used and attenuation parameter is introduced to reduce the influence of knowledge distillation on student model.Experimental results show that,when the source dataset and target dataset are both CIFAR100,the effectiveness of the proposed algorithm exceeds KD,Fit Nets,AT,RKD and Sim KD algorithms,and the accuracy is 0.49%,0.60%,0.38% and 0.24% higher than that of the second-best model in different model Settings.When the source dataset is Image Net dataset and the target dataset is Caltech101,CUB200,MIT67,Stanford Dogs and Stanford40 datasets,the algorithm in this thesis outperforms Fit Nets,AT,RKD and Sim KD algorithms.When the target dataset is Caltech101,the accuracy is0.56%,0.78%,0.54% and 0.40% higher than that of the second-best model in different model Settings.When the target datasets are CUB200,MIT67,Stanford Dogs and Stanford40,and the teacher model and student model are Resnet34 and Resnet18,the accuracy is 2.07%,2.46%,1.37% and 3.12% higher than the second-best model.