Research On Relationships Between Tensor Networks And Neural Networks

In the era of big data,tensor is an efficient way to express the large-scale data.And tensor networks(TN),which connected among tensors,can significantly alleviate the curse of dimensionality on large-scale data by employing higher-order structure informa-tion.On the other hand,nowadays neural networks(NN),as the mainstream research direction in the field of AI,have obvious superiority but have the problems of curse of dimensionality as well,like too large number of parameters,lower training speed and very redundant parameters.However,since the input,output and weight of NNs can be seen as special tensors,we can apply the theory,thinking,methods and techniques of TN to NN.Then the relationships between TN and NN can be built to improve the neural networks and to attempt to interpret NN from the aspect of TN.The main contributions of this paper are optimizing the computation of TN and proposing a new tensorizing neural network method,and they are concluded as follows:Above all,the diagrams of several classic tensor decompositions are studied and their pros and cons and correlations between them are analyzed,which are prepared for the following content.Next,a parallel probabilistic temporal tensor factorization(P~2T~2F)method is pro-posed for the problem that the probabilistic temporal tensor factorization(PTTF)model is difficult to speed up via parallelization.The core concept of the P~2T~2F is dividing each sequence operation on a large-scale tensor into some independent corresponding oper-ations on small sub-tensors,then using ADMM framework to speed up computing and also maintaining the ability of temporal effects of PTTF model.A new stochastic learn-ing algorithm is also designed to further improve the performance.Experimental results demonstrate that P~2T~2F is superior to contrast algorithms in accuracy and scalability.Lastly,”block-term decomposition based tensorizing neural networks”(BT-Net)is proposed to solve the problem that nowadays when compressing deep neural networks,the networks are either not efficiently be compressed or requiring a heavy fine-tuning process.BT-Nets mainly use the block-term layers,which posses an extremely few pa-rameters via block-term decomposition,to replace the fully-connected layers and convo-lutional layers for the sake of compressing the number of parameters.This paper points out that due to the”exponential representation ability”of the block-term layers,BT-Nets can preserve the representation ability of fully-connected layers and convolutional layers as much as possible;And due to the”commutativity”of block-term layers,the ranks of BT-Nets are not requiring a fine-tuning process.Experimental results demonstrate that BT-Nets can largely compress the number of parameters of a single layer while almost maintaining or even higher than the original performance and the total compression ratio is impressive as well.