Research On Multivariate Time Series Prediction Based On Random Project Neural Networks

Multivariate time series widely exist in the complex systems of natural, industrial, economic, energy and social areas. The research on how to effectively mine potentially useful knowledge from these time series, and build prediction models, not only can help us to comprehend and understand the actual complex systems, but also can provides theoretical support for control, decision and regulation. This dissertation will focus on multivariate time series prediction based on random project neural network, and use two newly proposed random project neural network:extreme learning machine and echo state network as prediction models to develop offline and online modeling methods that are suitable for multivariate time series prediction. It covers:1. The extreme learning machine prediction methods based on model selection and improved training algorithms are proposed. For the problem that the number of extreme learning machine hidden layer nodes is difficult to determine, extreme learning machine based on model selection algorithm is designed to select the optimal number of hidden layer nodes. For the ill-posed problem in extreme learning machine pseudo-inverse training algorithm, the trust region Newton training algorithm is introduced, which can achieve better prediction results than the pseudo-inverse training algorithm, and is particularly suitable for large-scale problem. Taking into account the discrepancy influences on prediction results of different samples, extreme learning machine with kernels based on instances weighted algorithm is established and different weights are applied to samples according to their prediction errors. In addition, in order to solve the problem of kernel function selection and kernel parameters optimization of extreme learning machine with kernels, multiple kernel extreme learning machine is developed, in which, several different types kernel functions or one type kernel functions with different parameters are combined to improve the nonlinear expresssion ability.2. The echo state network prediction methods based on improved training algorithm and hierarchical structure are proposed. The prediction error of noisy time series is decomposed into prediction model bias and input perturbation error, and it is noted that the tradeoff between these two items is the key of high precision modeling. Under the assumption that the noise is bounded, robust echo state network is proposed. For the problem that the regularization coefficient of support vector echo state machine is difficult to determine and its prediction accuracy is restricted by the single regularization coefficient, weighted support vector echo state machine is designed, in which the regularization coefficients are weighted and solved by the solution path algorithm. For complex multivariate time series prediction problem, hierarchical structure echo state network is constructed, multiple cycle reservoirs with jumps are used in the first layer extract dynamic features and the dynamic features are first preprocessed by feature selection algorithm and then modeled by extreme learning machine.3. The random project neural network prediction methods based on improved online training algorithms are proposed. Considering the multivariate time series online prediction problem, extreme learning machine based on online LM training algorithm is developed. Combined with the random hidden layer generation mechanism of extreme learning machine, the iterative calculation formulas of Hessian matrix and gradient vector are derived. For non-stationary time series prediction problem, extreme learning machine with kernels based on online sequential training algorithm is proposed. According to the Lagrange function of extreme learning machine with kernels, incremental learning algorithm and decrement learning algorithm are derived, and a fixed time window prediction strategy is designed. In addition, focusing on the lack of effective online training algorithm and ill-posed problem of echo state network, echo state networks based on online sparse training algorithm is designed. The L1 regularization term is added in the objective function to improve the property and sparsity of output weights, truncated gradient algorithm is used to online approximately solve the L1 regularization problem, and the effectiveness is demonstrated by theoretical analysis.