Structure And Parameters Optimization Of Echo State Network And Its Application

The field of RC has been growing rapidly with many successful applications. However, RC has been criticized for not being principled enough, namely the reservoir which is unlikely to be optimal because the reservoir connectivity and weight structure are created randomly. Reservoir constructionis largely driven by a series of randomised model building stages, with both researchers and practitioners having to rely on a series of trials and errors. Echo State Networks (ESNs), Liquid State Machines (LSMs) and the back-propagation decorrelation neural network (BPDC) are examples of popular RC methods.ESN is a recurrent neural network (RNN) with a non-trainable sparse recurrent part and an adaptable readout from the reservoir. Typically, the reservoir connection weights, as well as the input weights are randomly generated. ESNs have been successfully applied in several sequential domains, such as time-series prediction, non-linear system identification, speech processing, mobile traffic forecasting, stock price prediction and language modeling and according to the authors, they performed exceptionally well. However, ESN has some main drawbacks such as poorly understood reservoir properties, reservoir specification requires numerous trails and even luck as the reservoir connectivity and weight structure are created randomly beforehand. The random connectivity and weight structure of the reservoir are unlikely to be optimal and do not give a clear insight into the reservoir dynamics organization. This has raised the question of how to create optimal ESN for a given task. The main contributions of this dissertation are divided into two parts, optimizing of non reservoir parameters (input scaling parameters and output connections) and reservoir parameters (reservoir size and parameters). The main contribution of this thesis are as follows:(1) The input scaling parameters need to be adjusted in the multi-input ESN models. The influence of input scaling parameters are seldom investigated in the multi-input ESN models nowadays. A principal ESN designing theory is still lacking. The optimization of neural network systems involves a number of manual, tweaking or brute-force searching parameters, such as network size, input scaling parameters, and spectral radius. To create an optimal ESN, we proposed an ESN with SensitivityAnalysis Input Scaling Regulation (SAISR) and Redundant Unit Pruning algorithm (RUPA). In SAISR method, an ESN with sufficient reservoir and default input parameters is established. Then train the established ESN and calculate the sensitivity of each input variables. At last the optimal input scaling parameters is obtained by the input variables sensitivity. The new ESN is established by the optimal input scaling parameters. In RUPA, The correlation coefficient between each reservoir unit state is calculate firstly. The two most relevant internal neurons exhibit redundant relations. Then we prune out the redundant readout connections and repeat the pruning process until the expected error is accepted. A fed-batch penicillin cultivation process is chosen to demonstrate the applicability of the modified RC. The results show that the input scaling parameters have a more important influence than other parameters in ESN, and SAISR-ESN outperforms ESN without tuning. The RUPA method improves the generalization performance and simplifies the size of ESN.(2) The optimization of output connections of ESN is a feature selection problem. A better ESN with optimized output connections can be obtained by optimizing the output connections of ESN. The optimization of output weights connection structures is a discrete, high dimensional, nonlinear optimizing problem. In order to simplify the ESN and improve the generalization of ESN, we proposed an ESN with binary particle swarm optimization (BPSO). Because the output connections of ESN is a discrete feature selection problem, so BPSO has been used as a promising method for feature selection problems, First, we establish and train an ESN with sufficient internal units using training data. Then the initial population is generated randomly and the velocity and the position of each particle is updated by its fitness value. At last the ESN with the best particle is established. Finally, the performance of the optimized ESN is evaluated through test data. Results show that the global optimizing method BPSO outperforms the classical feature selection method LAR method and RUPA. Meanwhile the architecture in the ESN with BPSO is simpler than that of LAR and RUPA.(3) The reservoir which is unlikely to be optimal because the reservoir connectivity and weight structure are created randomly. How to design and create the reservoir is still an open question nowadays. The Simple Cycle Reservoir Network (SCRN) is a new form of ESN with deterministically constructed connectivity and weight structure and achieved comparable performances to standard ESN on a number of widely used time series benchmarks of different origin and characteristics. Despite many advances of SCRN, determining the most appropriate network size is still a question in SCRN. A reservoir that is too large may precisely fit the training data but may provide poor generalization, increase computational requirement. Conversely, a reservoir that is too small saves computational costs but may unable to solve the problem as we expected. Therefore, reservoir selection should consider both network complexity and generalization ability. We proposed two methods which are Simple Disjunction Algorithm (SDA) and Sensitive Iterative Pruning Algorithm (SIPA) to improve the SCRN. SDA starts with a proper reservoir firstly. Then weights rate of each internal neurons are calculated. Finally internal neuron which weights rate exceed a threshold are spread into two neurons. SIPA starts with a larger than necessary reservoir firstly, and then its size is reduced by pruning out the least sensitive internal units. The results show that pruning methods are more proper than constructing methods such as SDA. Although SIPA outperform the SDA methods, however SDA is able to keep the output weights small and to make SCRN more stable. According to the experimental results SIPA can outperform the, SDA on theperformance. However, SDA can make SCRN more stable.