Research On Cholesky Factorization Based Multiple Hidden Layers Extreme Learning Machine Algorithms

Extreme Learning Machine(ELM)is a new learning algorithm based on single hidden layer feedforward neural network,which is one of the important research directions in the field of machine learning.Due to its advantages of simple implementation,fast learning speed and less human intervention,ELM has been widely used in classification,regression and other learning problems.In recent years,inspired by the successful application of deep learning in many practical problems,the academic research on ELM has changed from single layer network to multiple layer network.Multi-layer extreme learning machine(MELM)is a new framework that integrates the characteristics of deep neural network having with multiple layers into ELM.How to optimize the number of layers and the number of nodes to build a deep stacked network is the primary issue to be considered in the study of MELM algorithm.In view of the huge network scale brought by multiple layer structure,how to design efficient numerical calculation methods to meet the requirements of calculation speed and accuracy is another important issue to study MELM algorithm.Starting from the key factors affecting the performance of ELM,we have done some exploratory improvement research on network structure design and network parameters optimization.In network structure design,the single layer is extended to multiple layers,and genetic algorithm,random enhancement and tentative pruning are applied to optimize the number of the hidden nodes.To overcome the difficulty of solving a large number of high-dimensional linear equations caused by the increase of layers in the process of network parameters optimization,Cholesky factorization is used as the main mathematical tool,and tailored strategies such as forced positive definite and Givens rotation transformation are proposed to find the inverses of higher-order matrices in incremental or decremental manners,so as to reduce cumulative errors and computational complexity,to improve the accuracy of the prediction model.The main contents of this dissertation are summarized as follows:(1)Research on multi-layer extreme learning machine based on forced positive definite Cholesky factorization.For the partial matrix which appears in the process of parameters optimization for MELM model,its semi-positive definiteness is proved.A forced positive definite Cholesky factorization strategy is proposed to determine the hidden layer parameters,which improve the condition number of matrix while forcing the matrix to be positive definite.Therefore,the convergence speed of MELM model is accelerated and the numerical stability of the modeling process is guaranteed.Moreover,in order to avoid the influence of manual intervention on prediction accuracy and computational stability of the model,genetic algorithm is proposed to determine the optimal number of hidden layers and the corresponding optimal number of hidden nodes,and particle swarm optimization is introduced to search the optimal input weights and hidden layer bias,so as to improve its generalization ability and computational stability.(2)Research on incremental multi-layer extreme learning machine based on Cholesky factorization.To design the MELM network structure,an incremental automatic determination method of hidden nodes based on enhanced random search strategy is proposed.In the network generation process,according to the principle of structural risk minimization,an optimal node is selected from randomly generated hidden nodes and added to the network one by one until the prediction error meets the accuracy requirements.To avoid the repetitive computation of inverse matrix and Moore-Penrose generalized inverse matrix involved in the process of network enlargement,a Cholesky factorization based strategy is proposed to update the connection weight matrix in an incremental manner,which can avoid the cumulative error caused by complete recalculation of hidden layers and make full use of the historical information in the network generation process to update the network parameters.(3)Research on pruned multi-layer extreme learning machine based on Cholesky factorization.Aiming at the structural redundancy of MELM network caused by irrelevant variables,Tikhonov regularization method is applied to measure the influence of the number of hidden nodes on the learning accuracy and generalization ability of the network model.It is proposed that the hidden nodes should be pruned gradually in the learning process until the sum of empirical risk and structural risk of the model changes significantly.In the learning process of pruning hidden nodes,based on the connection weight matrix information of previous generation,Givens rotation transformation is proposed to calculate Cholesky factorization factor,so that the current connection weight matrix can be updated rapidly in a decremental manner.(4)Research on online sequential multi-layer extreme learning machine based on Cholesky factorization.For the online learning problem with strong timeliness of data samples,an adaptive optimal network structure and parameters learning method is proposed,which introduces forgetting mechanism to reduce the influence of old training samples and highlight the role of new training samples.In addition,in order to ensure that the network structure and parameters are dynamically adjusted with the changing sequential training data in online learning,an adaptive increment-decrement update strategy for hidden nodes is proposed.In the increment-decrement process of hidden nodes,the historical information,stored in each iteration of learning is fully utilized,and the Cholesky factorization is used for adaptive incremental or decremental inversion of higher order matrices,avoiding repeated calculation.All the improved MELM algorithms mentioned above are tested using some benchmark.data sets and actual data provided by enterprises.The numerical results show that the improved MELM algorithms can achieve higher prediction accuracy and better generalization performance than the conventional MELM algorithm,and verify the effectiveness of the proposed improvement strategies.