Research And Implementation Of Feature Selection Algorithm Based On Ensemble Learning

In the recent years,with the cost reduction of calculation and storage,ensemble learning has become a hot direction in machine learning.Through the combination of models,the ensemble model can achieve a huge breakthrough in both computing power and calculation accuracy.According to the differences of the base learner or training data,the integration method is mainly divided into heterogeneous approaches and homogeneous approaches.At present,there are some dimension reduction idea which apply ensemble thinking into the modeling based on feature selection,but the base learner has been assigned the same weight in most ensemble feature selection algorithms.In fact,regardless of heterogeneous approaches or homogenous approaches,it will generate some different base learners,whose adaptability of training set is different.Therefore,to solve blemish of existing ensemble feature selection algorithm,we propose ensemble feature selection algorithms based on weight adjustment.For homogenous approaches,this paper proposes an ensemble feature selection based on soft-max function.We adjust the weight of base learner according to their adaptability of training set by soft-max function,and those base learner,whose adaptability of training set is good,will be assigned higher voting weight.Furthermore,this paper analyzes the generalization ability of the ensemble feature selection algorithm based on weight adjustment in theory,and we design some experiments,which validate that the new algorithm has better generalization ability than the ensemble feature selection without weight adjustment.For heterogeneous integration,this paper proposes an ensemble feature selection algorithm based on genetic algorithm.This new algorithm uses genetic algorithm to evaluate,iterate and optimize the weight vector of the base learner to obtain the optimal weight vector.According to the type of weight vector returned,this method is divided into two kinds——the ensemble feature selection based on optimal weight and the selective ensemble feature selection.The weight vector,returned by the ensemble feature selection based on optimal weight,is continuously,and the weight vector,returned by the selective integration feature selection,is discrete.Finally,it is verified by experiment that those two methods have better generalization ability than the ensemble method without weight adjustment.In addition,on the basis of the selection of ensemble feature selection based on genetic algorithm,this paper proposes an ensemble feature selection method based on particle swarm optimization,because genetic algorithm is not good at dealing with continuous problems and it has high computational complexity.Finally,we design some experiments and prove that this new algorithm is faster than ensemble feature selection based on optimal weight and it has better generalization ability than ensemble feature selection without weight adjustment.