Research On Chaotic Time Series Prediction And Reservoir Machine Learning Method

Chaos theory serves an important part in modern science and technology, which reveals the inherent laws governing complex system. A complex system is a nonlinear chaotic system, which exhibits the sensitive dependence on the initial conditions. Two such systems with however small a difference in their initial state eventually will end up with a large difference between their states. The objective of the Chaos theory is to explore the order and determinacy behind these phenomenons. It has been twenty years since scientists started the research work of time series prediction based on Chaos theory, and many results have been obtained. However, many prediction methods stay in laboratories, and ignore the uncertainty in practical situations. Meanwhile the access to accurate prediction model relies on machine learning tools, which are expected to be efficient in dealing nonlinearity, excellent in generalization and reliable in robustness. As an emerging machine learning tool for dynamical system, reservoir method has been shown the excellent performance in chaotic time series prediction. It draws many interests but the mysteriousness still exists. Based on the above observations, this paper will focuses on reservoir method and try to find new techniques for chaotic time series prediction and machine learning applications. It covers:1. Analyze and construct a reservoir prediction model for chaotic time series. Reservoir can be used as a good predictor for some chaotic systems, however, the predictor is in the absence of reasonable explanation and difficult to apply to noisy situations at present. Based on the problem, the approximation ability of this kind of recurrent neural networks is verified in this study, and the emphasis is placed on the problem of trajectory learning and initial state setting. The reservoir model structure is classified into three categories: general state-feedback structure, output-feedback structure and the feed-forward structure. The existing iterative predictor can be well explained in the perspective of output-feedback structure, and it is shown that this structure will lead to two difficulties in modeling chaotic time-series: error accumulation and underlying stability problems. To eliminate these problems, a direct prediction method based on general state-feedback structure is proposed, and it relates the prediction origin and horizon directly. The stability can be assured before the network training, and the prediction error does not accumulate since there is no output-feedback loop.2. Propose a regularized learning method for reservoir. The existing reservoir training method is ill-posed, which has many symptoms, such as continuous singular value spread, large condition number and output weights. Regularized learning method is then proposed to curve the difficulties in this study. Regularization can be realized by truncated singular value decomposition(TSVD) or penalty methods. In TSVD, the small singular values are discarded to improve the solution; In penalty method, the ridge regression is used to improve the matrix structure to be factorized. The computation problem is also addressed, and it is shown that the penalty method can be more efficient than the truncation method. In addition, the theory analysis is carried out to the noisy chaotic system prediction problem, the worst prediction error caused by noise is computed in the style of Error-In-Variables model. Given a perturbation bound of the noise, it is shown that the regularization learning based on penalty method can result in an optimal robust solution.3. Propose a novel nonlinear support vector machines(SVMs) without a kernel. Based on the similarity between kernel and reservoir, a kernel-free SVMs—Support Vector Echo-State Machines(SVESMs) is proposed in this study. Classic SVMs rely on kernel method to compute the inner product, by contrast, reservoir method allows the direct creation of nonlinear mapping by the mechanisms of echo state property. The computation in the reservoir state space is straight-forward and it is easy to realize the structural risk minimization principle and introduce different loss functions for various applications. Robust loss functions can be used in SVESMs so that the model is insensitive to outliers, which is very suitable for practical time-series modeling. SVESMs can work both in recurrent and feed-forward modes. In recurrent mode, SVESMs are easy to train and do not get trapped in local minimums, and the generalization ability is well supported by the statistical learning theory. SVESMs working in the feed-forward mode behave much like the classic SVMs in many ways, such as hyper-parameter searching and capacity controlling, and feed-forward SVESMs serves a bridge from feed-forward neural networks to SVMs.