Investigations Upon The Algorithms For Selective Ensemble Of Neural Networks

Ensemble learning has become a hot topic in the field of machine learning, and selective ensemble is attracting more and more attentions by the advantage of its applicability and combinability to many learning machines. This dissertation investigates the selective neural network ensemble by means of the theories and methods of relational fields, such as information theory and computing science. Some approaches are proposed to construct selective neural network ensemble with high performance, and then their working mechanism and preferences are discussed in details. The major contributions of this dissertation are emphatically stated as follows.Firstly, one category of ensemble methods based on the strategy of global optimization is carefully explored. Two kinds of powerful optimization tools - Ant Colony Optimization (ACO) and Partical Swarm Optimization (PSO) are employed to construct selective ensemble so that selective optimum neural network ensemble based on ACO and selective optimum neural network ensemble based on discrete Binary PSO (BPSO) are proposed. In BPSO-based approach, each candidate ensemble corresponds to a position of n-dimension 0-1 space and the goal of constructing optimum ensemble is achieved by particle optimization in discrete binary space. In ACO based approach, the pheromone reflects the accrracy of ensemble while the diversity heuristic information indicates the diversity of individuls. Both approches show perfectly predictive ability.Secondly, clustering-based selective algorithm for constructing neural network ensemble is investigated, where neural networks are clustered according to similarity and the most accurate individual network from each cluster is selected to make up the ensemble. The usage of traditional k-means clustering is limited due to its strict requirements in data distribution. Alternatively, Spectral Cluster (SC) has no prerequest on the global structure of data. So selective optimum neural network ensemble based upon spectal clustering is proposed to improve the predictivity accuracy of selective ensemble, in which mutual information is used to measure the diversity of neural network and the group relationships among data points are preserved as much as possible in a lower dimensional representation.Thirdly, an idea named "ensemble of ensembles (EoE)" is proposed. Different from ordinary neural network ensemble, ensemble of neural network ensembles is a two-layered neural network ensemble architecture and employs weighted neural network ensemble as individual of ensemble. The advantage of ensemble of ensembles lies in that individual diversity can be manipulated by adjusting the weights of weighted ensemble rather than the architecture or function of neural network. Two approches based on EoE idea named EoE-MIL (Ensemble of neural Network Ensembles based on Minimum Information Loss) and EoE-AI (Ensemble of neural Network Ensembles based on mAximum Independence) are designed and implemented. In EoE-MIL approach, neural networks are combinated weightedly by the eigenvectors of principal eigenvalues of the covariance to construct individual of EoE accrording to minimum information loss principle, and the diversity among individual of EoE is guaranteed by the linear independence of eigenvectors. In EoE-AL approch, Kullback-Leibler information distance is used to measure statistic independence of individual of EoE, and weighted combination of neural network by the eigenvectors of principal eigenvalues of the correlation matrix becomes the individual of EoE as well according to maximum independence principle. Both approches show the ability of reducing predictive error and that of selecting the model coresponding to particular problem.Furthermore, the diversity of the neural network ensemble investigated.The future work should include a generalized and deeper theoretical study, to explore new powful constructing algrithms, and to expand the applications to a wider scope.