Research On Key Problems Of Recurrent Neural Network Models

The arrival of the era of big data has promoted the innovation of data analysis meth-ods.Deep learning technologies based on neural network have made breakthroughs in many fields.Recurrent Neural Networks(RNNs)have played a leading role in the pro-cessing of continuous time series and are widely used in many fields of research such as natural language processing,computer vision,and speech recognition.However,in practical application scenarios,RNN still faces many problems such as complex model structure,time-consuming training,low computational communication ratio,difficulty in distributed training,high decoding delay,and difficulty in learning long-term depen-dencies.The dissertation focuses on the key issues of RNN's efficient training algo-rithm,scalable distributed training,fast decoding algorithm and long-term dependency learning.The main research contents include the following four parts.(1)For the problems of high data coupling and difficulty in parallelization in RNN training,we study the RNN training on GPU platform,the computational logic rela-tionship in RNN training is sorted out and the parallelization potential can be explored.A data decoupling pipelined time domain backpropagation(BPTT)algorithm is then proposed.Compared with the traditional algorithm,the pipeline algorithm divides the forward and backward computation process in RNN training into multiple parallel mod-ules,and uses multiple computational flows to independently load the computation tasks of each module,thereby improves the RNN training efficiency on the GPU platform.(2)Aiming at the problems of high bandwidth requirement and poor scalability in the existing RNN distributed training algorithm,a distributed RNN algorithm based on data parallel method is designed.The algorithm forms a token-like ring topology of computing nodes,to synchronize the information during training between computing nodes.Compared with the traditional master-slave algorithm,the proposed algorithm significantly reduces the communication load in the system.On this basis,using the gradient sparsity feature of the RNN model,the gradient value of the communication transmission between the computing nodes is thresholded,and the gradient value be-low the preset threshold is delayed to the subsequent iteration.The experimental results show that the distributed training algorithm can achieve near-linear lossless acceleration on multiple computing nodes when the threshold is set properly.(3)Aiming at the problem that the decoding delay is too high in the sequence clas-sification task of RNN model,a parameter compression method based on matrix low rank decomposition technique is proposed,which makes full use of the singular value distribution characteristics of RNN hidden layer weight matrix.The network weights are reconstructed in a low-dimensional manner,which effectively reduces the compu-tational and storage complexity of the network in the decoding process.Experimental results show that the parameter compression method can speed up the decoding speed of RNN.(4)For the gradient disappearance phenomenon in the long-term dependency prob-lem of RNN,the traditional RNN structure is extended,and a Lightweight Recurrent Unit(LRU)based on context information is proposed.Compared with the traditional RNN models,the context information stored in LRU hidden layer has further trans-mission distance in time domain,and have good interpretability.LRU is superior to traditional models such as LSTM in recognition of hundreds or even thousands of se-quences of learning tasks.