The Generalized Training Methods,Algorithms And Applications For Competitive Learning Neural Networks

The amount of data in the Internet is ever growing because the development of global informatization.Since the information volume is expanding faster than the computational pow-ers that humanity commands,the space complexity of machine learning algorithms is becoming more important than before.Meanwhile,the informatization is spreading into industrial areas other than computer sciences,with the development of the concept and technology of Internet of things and ubiquitous computing.There are a great amount of memory critical computation units in these areas that require low space complex algorithms.Online learning is a machine learning setting,where the machine learning model is acquired by a single pass scan of the training dataset,and it is more space efficient than the traditional offline methods,which need to store the entire dataset for training.Besides,there are various applications,such as network intrusion detection,short-term electricity requirement prediction and traffic flow prediction,where the training data arrives continuously and the problem scale is growing accordingly.In these applications,the model constructed by machine learning algorithms need to be updated continuously,and online algorithms are a major solution.Though online learning is space efficient,it is difficult to solve nonlinearity,semi-supervised learning and model parameter selection all at once,because of the single pass learning setting.Moreover,some online learning algorithms is computationally inefficient in updating the ma-chine learning models.However,the competitive learning has no such problems.Unfortu-nately,the competitive learning researches mainly focus on clustering.For other tasks such as regression,semi-supervised classification and dimensionality reduction,the advantages of nonlinear capability and efficient online learning of competitive learning can not be utilized.This thesis focuses on generalization of the competitive learning algorithms to these tasks.The main contributions are listed as follows.1.A kernel density regression framework is proposed to transform competitive learning clustering to regression algorithms.Online algorithms is either parametric or nonparametric.The parametric algorithms take a unified analytical assumption about the dataset,and it is dif-ficult to select the optimal model with best parameters under the online learning setting.The nonparametric algorithms have no such difficulty,however,the majority of them are unable to give smoothed predictions and suffer from compromised generalization abilities.To address the difficulties both in the parametric and nonparametric learning algorithms in online learning,this thesis proposes a nonparametric training method to construct a parametric model.By exploiting the distribution learning abilities of competitive learning neural networks,the joint distribution of explanatory variables and response variables can be modeled,and the regression model be obtained by density regression.Then,the regression function is deducted directly from cluster-ing learning results in this thesis.The complete framework is a feed-forward neural network model,the weights of the hidden layer of which is trained by the competitive learning neural networks.The advantage of this framework is that the local minima of the back propagation is avoided,while at the same time the smoothed prediction advantage of feed-forward neural network is kept.The proposed methods has constant space complexity and O(n)computation complexity.Experiments on six UCI datasets show that the proposed method gives comparable results to the mainstream offline methods.2.The optimal smooth parameter selection of kernel density regression is deducted by max-imum likelihood estimation.There is one smooth parameter in the kernel density regression framework,selection of which is difficult to reach optima in online learning,since the learning is finished in one pass scan.To solve this problem,we propose a maximum likelihood smooth parameter selection framework.Under the online learning setting,the optimal smooth param-eter selection of kernel density regression is solved by two strategies of maximum likelihood estimation.To solve the parameter selection of mixture models,a Bayesian inferences process is employed to transform the maximum likelihood function into solvable form.Then through derivation techniques,equations about the smooth parameter are constructed and solved.Be-sides,a semi-supervised learning process is also proposed to select the smooth parameters with the complete distribution information of the test dataset.The proposed framework selection is the optimal solution under the online learning setting,which is vital for the online kernel density regression,because the online algorithms is finished within one pass scan of data and leaves no room for parameter tuning.Experiments show that the proposed method is comparable to other online and offline algorithms,and it is parameter free.3.An inverse competitive learning paradigm is proposed for semi-supervised extensions of competitive learning neural networks.The original competitive learning neural networks,es-pecially the self-organizing neural networks are inefficient to handle the labeled datasets,which make them difficult to solve the semi-supervised learning problem,where the training dataset contains both labeled and unlabeled samples.To solve this problem,this thesis proposes the inverse competitive learning.Whenever there is a conflicting of labels,the neurons are moved in an inverse direction from the original competitive learning.Then the graph cut algorithm is employed to complete the semi-supervised learning.Moreover,a fast graph cut algorithm based on greedy label propagation is proposed.The main advantage of this framework is the robustness to noise in datasets and the ability to perform online semi-supervised learning under the manifold assumption.The experiments on the intrusion detection dataset show that the pro-posed method is more accurate than the mainstream offline semi-supervised learning methods even that it is online.4.A framework is proposed to transform clustering into dimensionality reduction.Com-petitive learning is used in combine for online dimensionality reduction learning.The online learning and nonlinear dimensionality reduction are always difficult to combine.With competitive learning at the core,this thesis proposes an online nonlinear dimensionality reduction framework.First,a data prototype is acquired by the semi-supervised competitive learning neural networks.Second,two similarity matrixes are fusioned into one by a quadratic programming process.Finally,the neurons are transformed into the lower dimensional space and other samples are embedded by a kernel smoothing procedure.Besides,a data visualization techniques is proposed exploiting the topology learning abilities of competitive learning.The proposed framework has the advantage of high space complexity comparing to existing nonlin-ear algorithms.Experimental results on the intrusion detection dataset show that the proposed method is a more efficient alternative to the offline semi-supervised learning algorithms.5.Short-term traffic flow prediction is implemented based on the ensemble of the proposed kernel density regression.Since the prediction results and weights of the individual online re-gressors are changing constantly with the time passage,the ensemble of such online regressors are always difficult.To address this difficulty,this thesis proposes a weight storing technique to construct a competitive regressor ensemble framework.By construction of weighted learning,the ensemble of kernel density regression is enabled.The proposed framework is a combina-tion of online learning and ensemble learning.Since the ensemble enables the combined use of competitive learning based regression and other algorithms,it is vital for making the proposed regression algorithms applicable in the various industrial and social applications.In the exper-iments,the proposed method is boosted by Adaboost and the results show that the proposed methods outperform mainstream methods such as support vector machine,decision trees and extreme learning machines etc.,and they have the parameter free advantage.