Knowledge Distillation Based On Multi-neural Network

In recent years,deep learning has been extensively employed in various domains of daily life and production,resulting in remarkable achievements.Nonetheless,the conventional deep learning models commonly require substantial computational and storage resources,which considerably restrict their feasibility on devices with limited resources,for example,mobile phones and embedded devices.Therefore,it is necessary to explore low-resource-dependent network models and learning methods.Due to the great potential shown in model compression,knowledge distillation and its research progress have attracted considerable attention from both academic and industrial communities.Nevertheless,contemporary methodologies possess some inadequacies as follows:(1)It is difficult to reuse knowledge distillation experience in cross-task situations.(2)The optimization of parameters for a lightweight network without any data is a challenging task.This paper focuses on knowledge distillation on medical image segmentation tasks and knowledge distillation on lack of training data,carries out research work in the following two aspects:For problem 1,a multi-teacher knowledge distillation method based on differential weighted modeling is proposed.The same knowledge distillation method has achieved good results in the classification task,but the performance of the application in the segmentation task may be greatly reduced.The reason is that classification tasks assign equal significance to all categories,while various image regions in segmentation tasks have differing degrees of importance.Existing studies have shown that the boundaries of objects in segmentation tasks are fuzzy and difficult to accurately segment.In order to solve this problem,this paper proposes a method of differentially processing the boundary of the segmentation,and combines multiple teacher models with different abilities to jointly guide the learning of the student model,and perform differential weighting according to the specific performance of each teacher model.Experimental results show that the lightweight student model can also output better segmentation results.Aiming at problem 2,a data-free knowledge distillation method based on multi-network joint is proposed.In the case of only the teacher model and lack of training data,it is a challenge to complete knowledge distillation.In the existing work,some need to retain some information of the training data when training the teacher model,and this method is relatively limited;some do not need any information of the training data,but there are defects in the algorithm.This paper proposes a method for multi-generator union.The main idea is to use two generators G1 and G2,where G1 is used to generate difficult samples,G2 is used to generate simple samples to assist G1,and the two jointly adjust the student model;prevent the student model from only pursuing performance on difficult samples and ignoring some simple samples,which leads to the decline of the overall performance of the model.Experiments are conducted on multiple public datasets,and the performance of the student model is close to that of the training data.Finally,with the multi-teacher knowledge distillation method based on differential weighted modeling,a simple medical image segmentation system is designed and developed,which can output segmentation results in time for pathological images uploaded by users.