| Recurrent neural networks (RNNs) have powerful computational abilities and could be used in a variety of applications; however, training these networks is still a difficult problem. One of the reasons that makes RNN training, especially using batch, gradient-based methods, difficult is the existence of spurious valleys in the error surface. In this work, a mathematical framework was developed to analyze the spurious valleys that appear in most practical RNN architectures, no matter their size. The insights gained from this analysis suggested a new procedure for improving the training process. The procedure uses a batch training method based on a modified version of the Levenberg-Marquardt algorithm. This new procedure mitigates the effects of spurious valleys in the error surface of RNNs. The results on a variety of test problems show that the new procedure is consistently better than existing training algorithms (both batch and stochastic) for training RNNs. Also, a framework for neural network controllers based on the model reference adaptive control (MRAC) architecture was developed. This architecture has been used before, but the difficulties in training RNNs have limited its use. The new training procedures have made MRAC more attractive. The updated MRAC framework is very flexible, and incorporates disturbance rejection, regulation and tracking. The simulation and testing results on several real systems show that this type of neural control is well-suited for highly nonlinear plants. |
