A Study Of Random Feedforward Neural Networks Based On Particle Swarm Optimization

Compared with feedforward neural networks trained by gradient based learning algorithms,random feedforward neural network(RFNN)has much faster learning speed and better generalization performance,which has been studied intensively and widely applied in many fields in recent years.However,the randomization of input weights and hidden biases in the network leads to two main defects in RFNN:(1)Compared with the feedforward neural networks trained by gradient based learning algorithms,more hidden nodes are required in RFNN,thus the increased complexity of the network reduces its generalization performance.(2)Non-optimal or non-approximate optimal input weights and hidden biases will make the output weights non-optimal or non-approximate optimal,which will eventually affect the performance of the network.Therefore,it is crucial to optimize the input weights and hidden biases in RFNN to enhance its performance.Compared with other meta-heuristic optimization algorithms,particle swarm optimization(PSO)algorithm has many advantages including fewer parameters need to adjust,no complex evolutionary operations,easy implementation,and requiring smaller evolutionary population.It has achieved good effects on many nonlinear optimization problems.In this dissertation,PSO algorithm is used to optimize single random feedforward neural network,the ensemble of random feedforward neural network and deep random feedforward neural network to overcome the deficiencies caused by the randomization of input weights and hidden biases,respectively.The main purpose of this dissertation is to improve the generalization performance of the three kinds of random feedforward neural networks mentioned above as well as to decrease their complexity.The main works in this dissertation offer new thoughts of enhancing the performance of RNN and improving swarm intelligence optimization algorithms,and the main contributions are listed as follows:1.As for the learning algorithms for random feedforward network,the randomization of input weights and hidden biases could lead to too many hidden nodes,poor generalization performance of the network.To overcome the deficiencies caused by the randomization,a novel optimization method for random feedforwardnetwork based on input-to-output sensitivity of the network and PSO(PSOIOS-ELM)is proposed in this dissertation.In the proposed method,the PSO is used to optimize the input weights and hidden biases by encoding the input-to-output sensitivity information of the network on the training samples.The input-to-output sensitivity information is used to adjust the individual previous best position of each particle and the global best position of the swarm to reduce the input-to-output sensitivity of the network with the precondition of improving the convergence accuracy.The PSOIOS-ELM algorithm could effectively reduce the number of hidden nodes,increase the robustness of the network,and improve the conditioning performance and generalization performance of the network.Experimental results on several regression and classification problems also verify the effectiveness of the proposed method.2.To overcome the shortcomings of traditional ensembles of random feedforward neural networks,an approach(DO-EOBELM)is proposed to establish the ensemble of random feedforward neural network based on double optimization strategy with considering four aspects including the generation of candidate RFNNs,the selection of member RFNNs,the optimization of ensemble weights,and the elimination of redundant member RFNNs.Firstly,orthogonal RFNNs are generated by generating orthogonal bases to improve the diversity of the candidate RFNNs,and some RFNNs with low input-to-output sensitivity are selected subsequently to establish the candidate member pool of the RFNNs.Secondly,considering both the classification performance and the diversity of the ensemble system,attractive and repulsive particle swarm optimization(ARPSO)algorithm is used to select members from the candidate member pool of the RFNNs to form the ensemble system.Thirdly,the ensemble weights of each member RFNN are optimized by ARPSO.Finally,the redundant member RFNNs with much lower ensemble weights are removed to reduce the complexity of the ensemble system with the precondition of convergence ability.Experimental results on functional regression,the classification of the benchmark data and gene expression profile data verify that the DO-EOBELM method could construct the ensemble of the RFNN with more compact structure and better generalization performance.3.Aiming at overcoming the defects of traditional deep RFNNs whose performance are affected by the randomization of the parameters of the auto encoders,a deep RFNN based on PSO(PSO-ML-ELM)is proposed in this dissertation.In thePSO-ML-ELM method,PSO algorithm encoding the input-to-output sensitivity information of the network is used to optimize the input weights and hidden biases of the RFNN related each hidden layer.By improving the performance of the auto encoder related to each hidden layer,the performance of the deep RFNN could be improved.After establishing the deep RFNN,the weights of the whole deep network are optimized by PSO to further improve the performance.Experimental results on different data sets verify that the PSO-ML-ELM method has a good balance between training time and generalization performance.4.The theoretical analyses of the PSOIOS-ELM and DO-EOBELM methods are given in detail in the dissertation.To reduce the computational cost of computing the input-to-output sensitivity values in PSOIOS-ELM algorithm,several kinds of simplified input-to-output sensitivity functions are deduced according to the monotonicity of derivative function of hidden activation functions in the RFNN.Thus,the training cost of the RFNN is reduced without decreasing the generalization performance of the RFNN.To solve the problem that the threshold value is difficult to determine when the DO-EOBELM algorithm removes redundant member random neural networks in the ensemble RFNN,the corresponding theoretical analysis on classification problem is presented,which provide a guide to how to determine the threshold.