Training Error And Sensitivity-based Feature Selection For Ensemble Of RBFNNs

In recent years,machine learning and artificial intelligence have been developed rapidly.They also get a wide range of applications in people's daily life which yield great impacts to our society.The task of feature selection is to select relevant features and discard irrelevant or redundant features.Then,machine learning is applied to learn models with more useful informative features for better performances.In recent years,ensemble learning combining outputs of multiple models usually outperform single models.Therefore,feature selection is also combined with ensemble learning to implement ensemble feature selection expecting to further improve the performance of machine learning.Most of existing ensemble feature selection methods focus on selecting feature subsets via a minimization of training error only.However,the key of machine learning is to improve the generalization capability of models,i.e.the capability of predicting future unseen samples accurately.Models with good generalization capability do not only achieve good training performance in training data but also remain stable when the input changes slightly.Over-emphasizing on the minimization of training error leads to overfitting and thus models are too sensitive with respect to small changes of input features.In this way,models cannot generalize well to future samples.In addition,multiple feature subsets are selected independently in many existing methods.However,the final classification model is a combination of base models trained by these feature subsets.There may be interactions among these base models,which may restrict any further improvement of the classification performance.In view of the above problems in existing methods,in this paper,an ensemble feature selection method TESEFS(training error and sensitivity-based ensemble feature selection)with RBFNN(Radial Basis Function Neural Network)based on training error and sensitivity is proposed.The training error and the sensitivity of an ensemble are used as two objective functions to select optimal feature subsets for all base classifiers by using the multi-objective genetic algorithm NSGA-III(Non-Dominated Sorting Genetic Algorithm III).Experimental results on 18 datasets show that the proposed method outperforms state-of-the-art methods and makes significant improvements in most cases.