Research On The Structure Design Method And Applications Of Recurrent Neural Network

Recurrent Neural Networks (RNNs) are more and more popular because of theirdynamics. There are some problems. The states of network and the learning algorithmare hard to convergence and their structures can not adjust by depending on the goalsystem. These problems limited their application and development. This paperfininshed some research work on designing the structure of networks, learningalgorithms and self-organizing algorithms based on the theory of dynamic systems.The main contents of the paper are as following:1) Design the structure of the networkFor balancing the approximating ability and stability of the network, this paperproposed a method for designing a stable recurrent neural network (Stable RecurentNeural Network, SRNN). Its structure splits the full connected hidden layer to twohidden layers. One of the hidden layer is a feed-back hidden layer and the other one isa self-organizing hidden layer. The feed-back hidden layer with many2×2blocksmakes the network have a weak stability conditions and wealth dynamics. Theself-organizing hidden layer can transform the information from different blocks,impove the approximating ability and organize the different dynamic subsystems towork. These make the network can adjust its structure depending on the tasks. Theexperiments prove that the network has a great approximating ability and a weakstalibity conditions.2) Design the training algorithm for SRNNThe training algorithm of the recurrent neural network is searching the globaloptimal point in the error surface actually. Before we design the training algorithm,we analysis the error surface of the recurrent neural network, from simple tocomplicated, and obtain the features of the error surface. Based on these characters,we designed the initialization method and training algorithm of SRNN. For theinitialization, we proposed a block-uniform design method which separate the errorsurface into some blocks and find the optimal block using the uniform design method.For the training algorithm, we proposed the improved gradient descent method whichadds a punishing item into the cost function of the old gradient descent method. Thisalgorithm makes the network have a great approximating ability and keep the networkstate stable well. All of these improve the practicality of the recurrent neural network.The experiments prove that the initialization method and training algorithm canconverge easily. It can improve the approximating ability of the network on thepromise that the states of network are stable.3) Design the self-organizing process of SRNN If the SRNN has less neurons, the network will have a weak approximatingability. If the SRNN has too much neurons, the network will has a complicateddynamics, then the states of network will be divergence easier and it has a highcomputing cost. Considering the features of the SRNN, we defined the dynamicsubsystem of the network in this paper. The network adjusts the structure by addingthe subsystems. Besides, this paper also proposed another self-organizing method toadjust the structure by activating the different subsystem when the network was usedto approximate different system.4) Research on the applications of SRNN in the nonlinear systemArtificial Neural Networks are widely used in the modeling and controllingprocess of the nonlinear system. This paper makes the Waste Water Treatment Process(WWTP) as a sample. We designed the soft-measuring modeling using the SRNN,control the dissolved oxygen by combining the SRNN and adaptive control and solvethe HT HJB equation based on the SRNN for the optimal control in the WWTP. Theyimproved the effluent quality and saved the energy consumption effectively.