A Study On Regularization Learning Algorithm For Noise Data

This dissertation is a study on regularization learning algorith-m for noise data, improving the correlation algorithms of neural networks with random weights(NNRWs). NNRW algorithm has fast learning ability and strong approximation ability. However, for noise data, the model does not have a stability and sparsity. In particular, it is not robust for outliers, even when the training set is too large, this method may not be solved.In this dissertation, we mainly study l2-l1 regularization algorithms for neural networks with random weights, the probabilistic learning algo-rithm for robust modeling using neural networks with random weights, the distributed approximate Newton-type approach on neural networks with random weights for large-scale data. The main contents are as follows:1. We propose l2-l1 regularization model of NNRW, aimed at taking into account the sparsity and stability simultaneously. The l2-l1-NNRW model does not have formal solution since the l1 norm is not differentiable. We mathematically derived the l2-l1-NNRW iterative algorithm on output weights by the knowledge of convex analysis and proved its convergence by an upper bound under the constraints on the activation functions, which in theory ensures that l2-l1-NNRW can train the networks effectively. The only constraint is that its activation function is bounded. Meanwhile, we gave the sparsity with respect to the solution to the l2-l1-NNRW and its stability. The experimental results indicate that l2-l1-NNRW has an ex-cellent effect, not only can avoiding the over-fitting phenomenon, but also achieving a sparse and stable solution.2. We propose a novel probabilistic robust NNRW(PRNNRW) algo-rithm in attempt to enhance the robustness of built NNRW using datasets with outliers. The key idea of our approach is to combine the l1 loss func- tion with the l2 regularization term according to the sparsity of outliers and compressive sensing theory. Under the assumptions with Laplace er-ror and Gaussian prior distribution, we mathematically offer a probabilistic interpretation of the robust model, which can be equivalently transformed to a probabilistic robust model by exploiting hierarchical representation of Laplace distribution, and the probabilistic robust model can be solved by devising a PRNNRW algorithm based on EM algorithm. The experimen-tal results demonstrate that the proposed PRNNRW algorithm can perform robustly with respect to the presence of outliers.3. We propose a new distributed approximate Newton-type NNRW (DANE-NNRW) algorithm, based on iterative method in an attempt to solve the model of NNRW for large-scale data problems. The large-scale training samples are sliced up into some small parts due to the inapplica-bility of NNRW algorithm for large-scale datasets. And for each part, we use a local learner model. The crux idea of our approach is to utilize a distributed approximate Newton method, which takes into account not on-ly the process on each local model but also the relationship between other local models. We mathematically derived the DANE-NNRW iterative al-gorithm on output weights in terms of Bregman divergence and Taylor’s formula and proved its convergence, which in theory ensures that DANE-NNRW can train the large-scale problems efficiently. The experimental results imply that the proposed DANE-NNRW algorithm can perform ef-fectively for large-scale data problems.