Research And Application Of Gradient Boosting-based Negative Correlation Learning Algorithm

The first work of this thesis is about research on ensemble learning. In the field of machine learning, we call the system or mode which has the ability to learn from the experience and knowledge as a leaner. Generally, training a strong learner costs much more than training a weak learner. The simple idea of ensemble learning is combining a batch of weak learners to get a strong learner. A common understanding is that the generalization performance of base learners and the di-versity among base learners are two keys to the success of an ensemble model. Some most useful ensemble learning algorithms, such as Bagging and Boosting, maintain the diversity implicitly during improving the performance of each base learner. Negative correlation learning (NCL) is a useful ensemble learning ap-proach for neural network ensembles, aims to train ensemble models by keeping the correlation among base classifiers as smaller as possible. NCL poses a correla-tion penalty term onto the loss function (i.e., the mean square error) of standard neural networks, which will be lager when the output of the neural network is correlated with that of other neural networks in the ensemble. Therefore, diverse classifiers would be trained explicitly given the loss of each neural networks is minimized.In this paper, a new NCL algorithm, GB-NCL is proposed, which replaces the base learners from neural networks to gradient boosting machine (GBM). The original NCL is only implemented with neural networks as the base learner, since it is easy to add a correlation penalty term to the original loss function of neural networks, and the result new loss function can also be minimized with standard neural networks’ training algorithm. Therefore, learning models which are not trained with an explicit loss function, such as tree-based methods, are not suitable to NCL directly. Obviously this is a big obstacle blocking NCL to be a more scalability ensemble learning method. We will explain why GBMs can be used in NCL and how to train them using NCL algorithm framework. We test our approach on some classification tasks. The experiment results show that NCL.GBM owns significant higher performance compared to the original GBM and NCL algorithms in most cases.The second work of this thesis is on hyperspectral image classification. Hy-perspectral image classification is a key problem of many hyperspectral image applications. Among the existing approaches for this problem, Active Learning (AL) and Semi-Supervised Learning (SSL) techniques have attracted a lot of at-tentions in recently years. However, AL usually require a lot of human label effort, while SSL, although is capable of assigning pseudo-labels to unlabeled data, may introduce incorrect pseudo-labels and deteriorate the performance of the classifier.To overcome these drawbacks, a novel approach, namely Regularized Col-laborative Active and Semi-Supervised Labeling (RCASSL), is proposed in this thesis. RCASSL can be viewed as an integration of AL and SSL techniques. On one hand, like many AL and SSL algorithms, it iteratively acquires labels or assigns pseudo-labels to unlabeled data, with which a Support Vector Machine (SVM) is trained in an iterative manner. On the other hand, it involves a novel validation strategy to verify whether a pseudo-labels correct or not. We use the proposed GB-NCL to implement this pseudo-label validation (PLV) strategy. At each iteration, only those pseudo-labels that have been validated as correct will be remained and used for updating the classifier. Empirical studies demonstrate that RCASSL can achieve higher classification accuracy than MCL-ECBD and RCASSL-NoPLV with the same amount of labeled samples. MCLU-ECBD is a state-of-the-art hyperspectral image classification algorithms. RCASSL-NoPLV is a modified RCASSL algorithm which removes the PLV. The compartment results between RCASSL and MCLU-ECBD indicate that introducing SSL can improve the performance of AL algorithm. The compartment results between RCASS-L and RCASSL-NoPLV indicate that the PLV implemented using GB-NCL can ensure the quality of pseudo-label data.