Research On Model Distillation Via Filter Knowledge

Knowledge distillation is a crucial technique for compressing models,allowing for the extraction of knowledge from complex teacher models to lightweight student models.However,the performance improvement of the student model is often limited due to differences between the teacher and student models.To address this issue,this thesis proposes a model distillation algorithm via filter knowledge,which considers the training accuracy requirements of the student model and optimizes filters based on research in knowledge distillation and filter knowledge.The main contributions of this thesis are as follows:1.Due to the high computational cost of filters caused by deep network training,previous model methods may prune important filters,and the accuracy of network model may not be excellent enough.To address this issue,this thesis proposes a model distillation algorithm based on fusion filter pruning.The algorithm compacts the network model by pruning the redundant filters that can be replaced,rather than the "relatively unimportant" filters in the norm.Additionally,model distillation is used to supervise the training of the network model,further balancing the accuracy problem caused by sparse parameters after pruning the filter and improving the model’s accuracy.Experimental results on several different types of visual classification datasets demonstrate that pruning models with the proposed model distillation algorithm based on fusion filter pruning can effectively reduce the complexity of the network model without substantially negative impacting its performance.2.The performance improvement of the student model is limited by the inherent gap between the teacher model and the student model,and the different outputs of the two networks in online knowledge distillation may cause performance degradation.To address this issue,this thesis proposes a new collaborative knowledge distillation via filter knowledge transfer.The algorithm proposes a collaborative learning framework that transfers excellent filter knowledge between networks and guides each other to achieve collaborative training.Moreover,by exchanging predictive knowledge between networks,it is possible to use the collaborative knowledge distillation of the student network to further improve its performance,even when a huge and complex teacher network does not exist.Experimental results demonstrate that the proposed algorithm can provide more feature extraction capabilities for learning compact models by transferring excellent filter knowledge,and collaborative learning can obtain more knowledge and experience from other networks.3.An image classification prototype system is designed and completed based on the model distillation method via filter knowledge.The system provides image classification services with a visual interface for users.The running and testing results of the system demonstrate that the proposed algorithm has superior performance and can handle different data types of visual classification tasks.This verifies the effectiveness of the proposed algorithm in this thesis.The proposed algorithm in this thesis is based on the model distillation via filter knowledge,which can further deepen the knowledge transfer between networks and effectively extract feature information from visual classification data to improve the performance of classification algorithm.From a practical application perspective,the proposed algorithm can not only effectively improve the differences between small and large networks and optimize the performance of small networks,but it can also be widely applied in computer vision,data analysis,and many other fields.