Research On Optimal Design Of Echo State Network And Its Applications

With the development of brain-like computing and artificial intelligence,artificial neural networks?ANNs?have been widely used.ANN is an intelligent information processing system which can simulate the structure and function of human brain.The weights of traditional ANNs are generally trained by gradient-based algorithms,however,gradient algorithms usually suffer from problems,such as complex training process,local optimal,gradient disappearance and gradient explosion phenomenon.In recent years,echo state network?ESN?has attracted wide attention due to its fast learning speed and good generalization ability,and has been successfully applied in various fields.ESN imitates the circuit structure of recursively connected neurons in the brain,it contains the input layer,the reservoir layer and the output layer.The core of ESN is a reservoir composed of randomly sparsely connected neurons.The input weight,reservoir weight and relevant parameters of ESN are generally randomly initialized.In view of the random initialization of the weights,we establish the weight initialization model of ESN,and solve the optimization problem of ESN for random weights.To solve the collinearity problem of ESN,the learning model of output weights is established,the collinearity problem and ill-posed problem are solved.To solve the random setting problem of reservoir structure,self-organized model is established and the structure design is solved.Aiming at the measurement problem of effluent ammonia nitrogen in the urban wastewater treatment process,the soft-computing model of ESN is established.Therefore,the research on optimal design of ESN can not only promote the development of theory itself,but also has high practical application value.The main research work and innovation points of this dissertation are as follows:1.The research of weight initialization on ESNIn order to solve the problem of random initialization of ESN for the input weights and the reservoir weights,an ESN based on weight initialization?WIESN?is proposed.Firstly,Cauchy inequality and linear algebra are used to determine the optimized initial weights interval.Secondly,we can obtain that the optimized initial weights are related to the input samples,the reservoir size and the reservoir state.They can ensure that the neuron outputs are located in the activation region of the sigmoid function.Lastly,after the weights are initialized,the ESN is trained.2.The research of output weights calculation of ESN based on adaptive second-order algorithmAiming at the ill-posed problem of ESN,an ESN based on adaptive Levenberg-Marquardt?LM?algorithm?ALM-ESN?is proposed.Firstly,before training the output weights,the input weights and the reservoir weights are initialized by the linear algebra method,and the optimized weight interval is obtained.Secondly,the LM algorithm is used to replace the classical linear regression method of training output weights.The damping term is selected adaptively,the adaptive factor is modified by the trust region technique.Lastly,the convergence and stability analysis are given.Comparing with other ESNs,the simulation results show that the proposed method has better prediction performance.3.The research of output weights calculation of ESN based on sparse regularizationSince the reservoir is very large,it may have redundant neurons,leading to the collinearity problem.An adaptive lasso?Least absolute shrinkage and selection operator?echo state network?ALESN?is proposed.Firstly,ALESN can indirectly prune redundant neurons,so as to automatically select important reservoir neurons and obtain sparse model with oracle properties.Adaptive lasso is essentially a convex optimization problem with l₁ constraint.Therefore,the same algorithm can be used to solve the adaptive lasso.Secondly,for the selection of regularization parameters,an improved Bayesian information criterion is proposed to select the optimal parameters.Lastly,the stability analysis of the improved model is given.Comparing with other ESNs,the results show that ALESN has better prediction performance and smaller output weight range.4.Structure design of incremental regularized ESNFor the structure design of ESN,an incremental regularized echo state network?IRESN?is proposed.Firstly,the sub-reservoirs are generated by the singular value decomposition principle,the singular value of sub-reservoirs weight matrix are all less than 1.Secondly,according to the residual or problem complexity,the sub-reservoirs are added to the network one by one,until the termination conditions are satisfied.The echo state property can be guaranteed without posterior scaling of the weights in the growing process.Thirdly,the regularization parameters are selected by leave-one-out cross-validation method.Lastly,the convergence analysis of IRESN is given.Comparing with other ESNs,the simulation results show that the appropriate network structure and the high prediction ability are obtained.5.Structure design of pruning modular ESNFor the structural design of ESN,a pruning modular ESN?PMESN?based on sensitivity analysis is proposed.Firstly,a modular ESN of several independent sub-reservoirs is constructed by singular value decomposition principle.Secondly,the network scale fitness is defined according to the sensitivity,and the reserved modular sub-reservoirs are determined by the network scale fitness.Lastly,in order to save the sample information of the deleted sub-reservoirs and eliminate the overfitting information,the input weights of the reserved modular sub-reservoirs are updated by means of the mean horizontal propagation of weight.The echo state property could be guaranteed without scaling the weight of the reservoir.6.A soft-sensing model of sparse Bayesian ESN for effluent ammonia nitrogenA soft-sensing model of sparse Bayesian ESN?SBESN?is proposed to measure effluent ammonia nitrogen in the wastewater treatment process.Firstly,the optimal values of the two hype-parameters introduced to SBESN are determined by the II-type maximum likelihood estimation,and the output weight are related to the independent priors,so the complexity can be effectively controlled.If the output weight is 0,the corresponding reservoir neurons can be pruned,leading to a compact structure.Lastly,the trained SBESN model is used for the soft-computing of effluent ammonia nitrogen.The experimental results show that SBESN can realize the rapid and accurate measurement of effluent ammonia nitrogen.