Incremental Learning Based On Neural Networks Ensemble

Incremental learning is an important domain in machine learning. In incremental learning, the whole data set is not available in a lump and we can only get a part of the whole data set every time. The learning task is to learn new knowledge from new data sets with the previously learned knowledge preserved without accessing the previously data sets. In this study, incremental learning based on neural networks ensemble is investigated.First of all, a simple case of incremental learning was investigated and a method based on Negative Correlation Learning (NCL) and selective ensemble, i.e., Selective Negative Correlation Learning (SNCL), was proposed. In SNCL, when new data set comes up, a copy of the previous ensemble was used to learn the new data set by NCL. In this way, the ensemble could learn well from the new data set due to the use of NCL and more or less preserve the previously learning knowledge since the copy of the previous ensemble was used. After learning the new data set, the new ensemble was combined with the previous ensemble and then a selecting process was employed to keep the size of the ensemble fixed. In the experimental studies,3real world data sets from UCI machine learning repository and2real world bioinformatics data sets were used to demonstrate the advantage of SNCL.After the research of SNCL, we realized that the class imbalance may easily occur in incremental learning. To make a better study in class imbalance incremental learning, class imbalance problems in normal machine learning situation were investigated firstly. We discussed the characteristics and challenges of class imbalance problems and educed the idea of dynamically selecting examples for training. A method based on Multi-Layer Perceptrons (MLP), i.e., Dynamic Sampling (DyS), was proposed. In DyS, MLP with sequential learning mode was used as the base learner. During the training process, the examples that were fed to MLP were dynamically selected to update the MLP according to the real-time status of the MLP learned about the examples. The selection mechanism considered not only the situation of class imbalance but also the difficulties of the examples being correctly classified. In this way, DyS could emphasize more on the minority classes and the difficulty examples. In the experimental studies,18multi-class imbalanced data sets from UCI machine learning repository were used to compare DyS with other relative methods and the results demonstrate the success of DyS.Finally, we investigated class imbalanced incremental learning based on the above studies. The cases of class imbalance in incremental learning include:the class distribution of the whole data set is imbalanced, and the class distributions of data subsets are also imbalanced; though the class distribution of the whole data set is balanced, the class distributions of data subsets can also be imbalanced. There are also some new classes or classes’loss in the new data subsets. We proposed a new method, i.e., Selective Further Learning with Hybrid Ensemble (SFLHE). In SFLHE, two kinds of classifiers, i.e., MLPs and Naive Bayes (NB) were used as the components of the ensemble. A group of impact weights (with the number of the classes as the length) was updated for every individual of the ensemble to indicate the’confidence’of the individual learning about the classes. The weights affect the outputs of the ensemble by weighted average of all individuals’outputs. The training of MLPs and NB considered class imbalance so that the ensemble can adapt the situation of class imbalance. In the experimental studies,3synthetic data sets and10real world data sets from UCI machine learning repository were used to simulate the incremental learning process. The results have shown that SFLHE could effectively address class imbalanced incremental learning. The further analyses have also shown that SFLHE could combine the advantages of both MLPs and NB to make a better model.