Training general dynamic neural networks

Scope and method of study. The purpose of this work was to develop gradient-based algorithms for training general dynamic neural networks. The first step was to define a general framework that can be used to represent a large class of dynamic neural networks: the Layered Digital Dynamic Network (LDDN), where many popular dynamic networks can be represented by the LDDN framework. The second step was to derive general procedures for computing the gradients for the LDDN. The key results of this research are two different algorithms for computing gradients and two different algorithms for computing Jacobians for the LDDN: the general Forward Perturbation (FP) algorithms and the general backpropagation-through-time (BTT) algorithms. The third step was the development of gradient-based optimization algorithms that are well suited for dynamic network training. This research analyzed the error surfaces for some simple dynamic networks and presented a newly discovered feature of these error surfaces: spurious narrow valleys that can trap optimization algorithms. Based on the analysis of these spurious valleys, we presented modified training algorithms that show improved performance on dynamic networks.; Findings and conclusions. There are two main contributions of this research. The first contribution is the development of two algorithms for gradient computation for a general class of dynamic network: the Layered Digital Dynamic Network. This research presented the Forward Perturbation algorithm and the BackpropagationThrough-Time algorithm. Each algorithm has a version for gradient-based optimization algorithms and Jacobian-based optimization algorithms. These two main algorithms have been introduced in the past by other authors, but this research has derived the complete equations for arbitrary LDDN's. The second principal contribution of this research is the discovery of a key characteristic of the error surfaces of dynamic networks: spurious narrow valleys that can trap optimization algorithms. This study analyzed these error surfaces and proposed new procedures that provide improved training for dynamic networks. This research also demonstrated the application of the proposed algorithms to problems in control systems and nonlinear filtering and also compared the memory, the speed and the computational complexity characteristics of the FP and BTT algorithms.