The Application Of Tensor Decomposition In Neural Network Model Compression

As the most popular direction in the field of machine learning,neural network has a wide range of applications in image recognition,natural language processing and other fields.It has incomparable learning ability of classical machine learning algorithms,but correspondingly,it needs strong computing power and storage capacity.This is why the classical model of neural network,M-P model,was proposed by McCulloch and Pitts in 1943,However,in the 21st century,especially in the last decade,it is one of the reasons why people pay attention to it.At that time,there was not a large amount of data,and there was not enough powerful processor and memory.It can be said that neural network is a learning algorithm driven by data and hardware,This hinders the application of neural networks to small computer systems,such as distributed systems.Therefore,it is a promising field to compress the learnable parameters of neural networks.In this paper,we mainly use the existing tensor decomposition method to compress the learnable parameters of neural networks.This paper first introduces the basic knowledge of tensor and its operation,tensor decomposition and so on,and gives the implementation of Kronecker product,strong Kronecker product,tt-svd,qtt-svd in Python,so as to make full use of NVIDIA’s GPU acceleration and pave the way for later applications.Furthermore,a method of CNN model compression is proposed.The first part is the compression method of linear layer,which is the core content of this paper,because in the common neural network,the learnable parameters of linear layer are much more than that of convolution layer,and the main methods used in this part are QTT and QCP compression;The second part is the convolution layer compression method.Compared with the linear layer,the convolution layer compression method is less,because it has a small number of learnable parameters,but it has a great impact on the accuracy of the model.The main method used in this part is CP decomposition.Then the model compression effect is verified on cifar10,stl10 and other classic data setsFinally,the application of model compression technology in computational aided drug design model is given.The results of numerical experiments show that the CNN model obtained by tensor compression technology not only has good model compression effect,but also can effectively reduce the loss of the model,The advantage of QCP scheme is also given:it still has better compression effect than qtt scheme when the rank is larger.This is the fundamental reason why this scheme is proposed in this paper...