| Although the traditional Deep Neural Networks(DNN)have achieved excellent results in various fields,the huge amount of parameters makes it difficult to migrate to resource-constrained platforms such as mobile devices,and it also causes inefficient training.The tensor ring models have low-rank structures and can represent large spaces with fewer parameters,so it can be utilized to compress traditional deep neural networks.In order to alleviate the problem of parameter redundancy and improve the ability of the tensor ring models,we will carry out the following two works by combining the tensor ring and neural network:1)For the problem of the excessive parameters of the Recurrent Neural Network(RNN),we propose a more compact tensor ring model to reduce training resource consumption.2)Aiming at the problem of large fluctuations in the Tensor Ring Convolutional Neural Network(TR-CNN),we design a new initialization method to stabilize the training process.The RNN is good at processing sequence data.However,when dealing with high-dimensional data such as video,the input-hidden layer will expand dramatically,resulting in parameter redundancy,which usually leads to a bad result.To solve this problem,we propose a Tensor Ring Recurrent Neural Network(TR-RNN),which decomposes the input-hidden layer into the form of a tensor ring to reduce the model parameters.Compared with the Tensor Train Recurrent Neural Network(TT-RNN)proposed by Yang et al.,TR-RNN is not much sensitive to hyper-parameters,and has a more simple structure.The innovations of this work are as follows:1)We introduce tensor ring decomposition into the RNN firstly;2)TR-RNN achieved 34192 times compression rate,nearly double the TT-RNN,and at the same time reached the highest accuracy;3)By using features extracted from pre-trained models,TR-RNN can still reach the level of competition with the-state-of-the-art uncompressed models.While the TR-CNN achieves good results,it also reveals that the network is not stable enough and often explodes or diffuses.Therefore,from the perspective of controlling the variance of data flow,we propose an initialization method,which is suitable for TR-CNN,to smooth the data flow and gradients and improve the stability of the network.The new initialization contributions are as follows:1)Smooth data flow and gradient transmission to constraint excessive fluctuations in the network;2)Improve network performance and achieve higher accuracy;3)It is flexible to set up and adapt to various ranks of tensor ring models. |
