Extended Kalman filter-based pruning algorithms and several aspects of neural network learning

In recent years, more and more researchers have been aware of the effectiveness of using the extended Kalman filter (EKF) in neural network learning since some information such as the Kalman gain and error covariance matrix can be obtained during the progress of training. It would be interesting to inquire if there is any possibility of using an EKF method together with pruning in order to speed up the learning process, as well as to determine the size of a trained network. In this dissertation, certain extended Kalman filter based pruning algorithms for feedforward neural network (FNN) and recurrent neural network (RNN) are proposed and several aspects of neural network learning are presented.; For FNN, a weight importance measure linking up prediction error sensitivity and the by-products obtained from EKF training is derived. Comparison results demonstrate that the proposed measure can better approximate the prediction error sensitivity than using the forgetting recursive least square (FRLS) based pruning measure. Another weight importance measure that links up the a posteriori probability sensitivity and by-products obtained from EKF training is also derived. An adaptive pruning procedure designed for FNN in a non-stationary environment is also presented. Simulation results illustrate that the proposed measure together with the pruning procedure is able to identify redundant weights and remove them. As a result, the computation cost for EKF-based training can also be reduced.; Using a similar idea, a weight importance measure linking up the a posteriori probability sensitivity and by-products obtained from EKF training is derived for RNN. Application of such a pruning algorithm together with the EKF-based training in system identification and time series prediction are presented. The computational cost required for EKF-based pruning is also analyzed. Several alternative pruning procedures are proposed to compare with EKF-based pruning procedure. Comparative analysis in accordance with computational complexity, network size and generalization ability are presented. No simple conclusion can be drawn from the comparative results. However, these results provide a guideline for practitioners once they want to apply RNN in system modeling.; Several new results with regard to neural network learning are also presented in this dissertation. To provide a support for the use of recurrent neural network in system modeling, the approximate realizability of an Elman recurrent neural network is proved. It is also proved that FRLS training can have an effect identical to weight decay. This provides more evidence showing the advantages of using FRLS in training a neural network. Another theoretical result is the proof of the equivalence between a NARX model and recurrent neural network. Finally, a parallel implementation methodology for FRLS training and pruning on a SIMD machine is presented.