A Study On Some Distributed Learning Algorithms

This dissertation is a study on some distributed learning algo-rithms.On the one hand,we improve some classical algorithms under the distributed framework,and on the other hand,we propose some ideas to perfect the original distributed network.Some classical algorithms,such as neural networks with random weights(NNRWs),which has fast learning ability and strong approximation ability,can analyze the small data effi-ciently.However,when faced with a complex high-dimensional data,or data stored on more than one machine,these algorithms calculation would become slow,and even can not solve.Based on these problems,in this paper,we study and discuss distribut-ed learning algorithms based on big data,mainly including distributed learn-ing algorithm based on NNRWs with l1 regularization(l1-NNRWs),the s-parse factorization algorithm of nonnegative matrix in distributed network,and the distributed factorization algorithm based on big nonnegative ma-trix.The main contents are as follows:1.We propose the l1-NNRWs algorithm based on average consensus in a distributed framework,which aims to combine the advantages of the sparsity in the l1-NNRWs algorithm and the iterative solution.Although the NNRWs with l2 regularization has the formal solution,when in the face of big data,it is difficult to calculate the Moore-Penrose,and easily lead to the over-fitting phenomenon.However,the l1 norm is not differentiable,which means it only has the iterative solution.The proposed algorithm is described here:we firstly divide the high-dimensional data sample into a number of sample data sets,for each sample set,set up the l1-NNRWs al-gorithm as a local model,then projection gradient method is used to solve these local models,finally get all local parameters to interaction and itera-tion to gain the limit value,which can make sure that per local model has the same paraments.The convergence of algorithm has been proved.And experimental results indicate that the proposed algorithm is efficient when analyze the big data,and its solution is sparse,which can be easily stored.2.According to the sparse decomposition of nonnegative matrix prob-lem,we put forward a nonnegative matrix sparse decomposition algorith-m based on distributed network,trying to solve the sparse decomposition problem of large-scale matrix,and the data storage problem after decom-position.The key point is to put the idea of the distributed network on the sparse decomposition of nonnegative matrix.So we can avoid the for-mer distributed method which need get all parameters in each node of dis-tributed network,and then synchronously solve the limit value of these paraments.Firstly,we divide the large-scale matrix in columns,and then improve the existing matrix decomposition algorithms which are used to solve small-scale matrix.The important point is combining the interactive ideas of distributed network to sparse decomposition algorithms of matrix,finally a novel non-negative matrix sparse decomposition algorithm based on distributed network is proposed.The convergence is proved theoreti-cally by us.Experiments show that the proposed algorithm for large-scale nonnegative matrix sparse decomposition has better accuracy and efficien-cy.3.For the problem of large-scale nonnegative matrix decomposition,we propose a distributed learning algorithm of color,making the original distributed network has the characteristics of parallel computing,speed up the calculation efficiency.We firstly use the Welch Powell to color the connected distributed network,and split large-scale nonnegative matrix in-to smaller pieces by its columns,then use parallel computing to solve small matrix decomposition model in the nodes with same colors,finally consider the interaction of local model between nodes.