Advanced architecture and training algorithms for recurrent neural networks

Recurrent neural networks (RNN) attract considerable interest in computational intelligence fields because of its superior power in processing spatio-temporal data and time-varying signals.; Traditionally, the recurrency of a neural network occurs between input samples along the time axis. The simultaneous recurrent network (SRN) extends the recurrent property to the spatial dimension. Presenting the feedback information with the same input vector to the network illustrates the transient properties of the system, which helps to trace the error propagation and facilitates the training at last. Backpropagation through time and extended Kalman filter are proved to be suitable gradient-based training algorithms for RNN.; Population based algorithms provide an alternative solution for RNN training when the gradient information is costly to obtain, or even unavailable. The evolutionary training employs stochastic search algorithms to find a near-optimal solution. Particle swarm optimization (PSO) and evolutionary algorithm (EA) are two successful approaches among many variants of evolutionary training methods. Despite utilizing similar evolution procedure, PSO and EA concentrate on different search techniques during the evolution, which leads to a faster convergence. In PSO, particles are also sharing the search information through a global best solution. While in EA, the selection pressure forces each individual to find a better position for survival; and the mutation factor helps the population to maintain a good level of diversity. An innovative hybrid PSO-EA algorithm discussed in this dissertation inherits the advantages of both PSO and EA, i.e., the cooperation and competition, by integrating evolutionary operators, such as selection and mutation, into the standard PSO.; The architecture and training methods discussed above have achieved good performance in solving the challenging real world applications, such as car engine classification, game of Go and time series prediction.