Research On Gated Units In Recurrent Neural Network

With the development of science and technology,numerous data need to be analyzed through neural networks.Most of these data have timing or context dependencies such as texts,pictures,videos and so on.Recurrent neural network can use its own characteristics to deal with these data well.As a part of the deep learning field,traditional RNN still has many deficiencies in dealing with complicated problems,such as maintaining good learning effect in deep network training or further improving the training speed and accuracy.To address the above two problems,this paper proposed several improved models of recurrent neural network with gate structure based on LSTM and GRU.(1)Based on the problem of slow convergence and poor training effect in deep neural networks,this paper proposed a cross-layer gate structure unit(CGU)which can connect information between layers.The data between different layers can also be selected and ignored as LSTM and GRU during the process of reverse gradient descent,improving training effect in deep neural networks.(2)Based on the problem of how shallow neural network can further improve the training speed and accuracy of the model,this paper combined GRU and Minimal Gated Unit(MGU),and proposed two Simplified Gated Unit(SGU)),SGU-u and SGU-m.By reducing the number of gates and changing the connection of gate structures,SGUs try to reduce the amount of parameter training in the gate structure learning process,and improve the training speed of the model while maintaining efficient work of the rest gate structures.In view of the above improved gate structure,this paper selected the MNIST image recognition,PTB language text data sets,and Quick Draw data sets,and compared the improved model with the traditional RNN,LSTM,GRU and other related models in literature.The experimental results showed that the gate structure based on CGU can be applied well in deep neural networks.The simplified gate structure based on SGU-u can also obtain higher training speed than MGU and GRU,while the SGU-m model performs poorly,which indicates that different gate structures have different functions and importance during the training process.