Research On Some Key Problems Of Recurrent Neural Networks

Recurrent Neural Network(RNN)is a basic artificial neural network in the field of deep learning,and has been widely applied as a memory model for processing sequential data.How to deal with the gradient vanishing and exploding problems is always the key and the difficult point in training RNNs.For a long time,many methods have been proposed to solve the gradient problems in RNN training,and new RNN variants are constantly proposed trying to handle longer sequences.Although many classical models have been proposed,it remains a challenge to enable RNN models to capture and learn longer dependence in sequences while quickly responding to short-term changes.In this study,the relevant knowledge of deep learning,neural network and gradient back propagation algorithm is taken as the theoretical basis,the divide-and-conquer strategy for different scale dependence is taken as the research direction,and the design ideas of various types of RNN are combined to solve the problem that it is difficult for RNN to learn long-term and short-term dependence at the same time.In this thesis,several typical RNNs are well studied,and the key ideas and design motives are extracted.On the basis of these,two new types of recurrent neural network structures are proposed:(1)In this thesis,Dual Recurrent Neural Network(DuRNN)is proposed based on the distribution law of different scale dependence in multilayer RNN.DuRNN based on the theory of dependence distribution tendentiousness in multilayer RNN,using the divide-and-conquer strategy,makes the long-term and shortterm dependence captured by the recurrent connections of different layers and further improves the effect of dependence learning.Compared with LSTM,DuRNN can adapt to longer sequences while quickly responding to short-term changes.Comprehensive analyses are carried out on the proposed recurrent neural network,and several representative experiments are performed to verify this model.(2)In this thesis,Bidirectional Independent Recurrent Imputation for Time Series(BIRITS)is proposed,which combines the characteristics of Bidirectional Recurrent Imputation for Time Series(BRITS)and Independent Recurrent Neural Network(Ind RNN).BIRITS uses the bidirectional connecting strategy in BRITS to make up for the insufficient interaction between neurons in Ind RNN.And it enables long-term and shortterm dependence to be learned in temporal and non-temporal directions respectively,which improves the performance of the network in complex tasks.Two proposed recurrent neural network structures divide and learn the dependence in sequences in different ways and to different degrees,but both improve RNNs' capability of learning sequential information.The two networks can also be used in combination to create synergy that can achieve better performance.