Research And Implementation Of Model Compression Method Based On Knowledge Distillation

Deep neural networks have achieved ideal results in various fields of computer vision.Deep learning algorithms based on convolutional neural networks can be applied to various tasks under the training of large-scale data.However,existing neural network models not only have high computational complexity but also require a large amount of memory space for inference,which seriously hinders their deployment on hardware devices with limited computing resources or strict latency requirements.Therefore,how to reduce the computational cost of the neural network on the premise of high accuracy is the key to the implementation of deep learning algorithms.This thesis investigates related algorithms in model compression in deep learning and researches knowledge distillation,a classic compression algorithm.The main work and contributions are as follows:(1)Aiming at the problem that it is difficult to restore the network accuracy after pruning,we propose a Progressive Feature Distribution Distillation(PFDD)algorithm.It is based on a progressive training strategy that is efficient for matching feature distributions between compressed and original networks.Thus,it can notably exploit both external information from samples and internal information from the network,were using a small proportion of the training dataset can yield considerable results.Experiments on various datasets and architectures demonstrate that our distillation approach is remarkably efficient and effective in improving compressed networks’ performance while only 1%of original samples have been applied.(2)Focusing on the difficulties in applying knowledge distillation in super-resolution,we propose a distillation framework based on autoencoders.When traditional knowledge distillation is used in superresolution networks,the transmitted knowledge is too noisy,which hinders the training of super-resolution networks,resulting in poor performance.Our framework uses the encoder-decoder structure to capture highfrequency information in high-resolution images and then passes it to the super-resolution network through feature distillation,which improves the reconstruction quality of the network.The experiments show that the super-resolution network trained by our method can improve nearly 0.3dB in double super-resolution reconstruction and nearly 0.2dB in quadruple super-resolution reconstruction on each standard super-resolution dataset.