Research On L₁ Regularized Extreme Learning Machine Based On Alternative Direction Method Of Multipliers

With the improvement of computer performance and the rapid development of neural networks,avoiding over-fitting problems in feed-forward neural networks(FFNNs),choosing suitable network structure and obtaining better parameters have been the main issues of research.For high-dimensional nonlinear system regression and data classification problems,an l1-regularized Extreme Learning Machine algorithm based on the alternating direction multiplier algorithm(ADMM)is proposed in this paper.The main research results of this thesis are described as follows:1.An l1-regularized-ELM based on ADMM is proposed.Combined with PLS,PLS-l1-ELM is proposed and in order to improve the network model accuracy,a hybrid prediction model is proposed.By adding l1-regularization,the sparse model can effectively avoid the over-fitting problem caused by large number of hidden layer nodes and can enhance stability and generalization of neural networks.Since the employ of l1 regularization penalty in objective function,some problems such as non-differentiable gradient is raised.Solving the optimal value within allowable computational complexity by using conventional convex optimization methods is difficult.However,ADMM can efficiently solve the convex optimization problem of combinatorial objective function with complex properties.ADMM has the common characteristics of dual decomposition and augmented Lagrangian.Therefore,ADMM can perform distributed computing on the combined objective function.Experimental simulation results show that the proposed method has good performance.Convergence proof of l1-regularized-ELM is given by ordinary differential equation(ODE)method.In order to further improve the robustness and accuracy of the prediction model,a hybrid prediction model is proposed.This model contains three sub-models:PLS,l1-regularized-ELM and PLS-li-ELM,and the sub-models are connected by linear weights learned by principal component regression(PCR).PLS,l1?regularized-ELM,PLS-l1-ELM and hybrid prediction model are applied to the same soft sensing problem and experimental results indicate that hybrid prediction model has better generalization and robustness.2.Online ADMM-based Neural Networks for Sparse Supervised LearningWith the increment of amounts of data,dimension of data becomes larger and larger.It is easy to fall into "dimension disaster" and there are usually many noise or useless information in real data.To find a quick and accurate method for online modeling and classification is very important.We deduce the online ADMM based on the idea of Recursive Least Squares(RLS).Improved ADMM can reduce algorithm complexity and realize online learning.Experimental results show that the proposed method can achieve good performance in regression analysis and multivariate classification problems.