Distance Weighted Support Vector Regression Recursive Feature Elimination

With the advent of big data era,the numbers of samples and the dimensionality of feature have significantly increased.In many machine learning fields,there are a large number of irrelevant features existing in original feature list,leading to increase the computational complexity and affect the generalization of machine learning model.However,feature selection can avoid overfitting,reduce the complexity of model,which decrease the time and space complexity.Therefore,feature selection has significantly importance to study for machine learning fields.Feature selection can be regarded as a crucial preprocessing before conducting model,which is a necessary technique to improve the performance of learning model.There already exist many feature selection methods,but many of them are generic in the sense that they do not consider the particularity of different machine learning tasks.However,some feature selection methods do consider specific machine learning tasks and integrate these tasks into the feature selection process,e.g.,the wrapper methods,among which Recursive Feature Elimination is a commonly used oneIn this research,we specifically consider the application of feature selection in regression problems.Recursive Feature Elimination achieves good performance of feature selection,and has a good ability to deal with high-dimensional data,thus we decide to study Recursive Feature Elimination as our research direction.We first improve the performance of the original Support Vector Regression and develop a novel regression model called Distance Weighted Support Vector Regression,which is robust to noise and not easily affected by the distribution of the boundary data.Therefore,Distance Weighted Support Vector Regression could have better fitting performance.Then we implement a specific Recursive Feature Elimination process based on Distance Weighted Support Vector Regression,which is insensitive to the change of parameter values and robust to noise.The proposed Distance Weighted Support Vector Regression Recursive Feature Elimination ranks the features according to their contributions to Distance Weighted Support Vector Regression at each iteration and eliminates the less important features progressively.The performance of Distance Weighted Support Vector Regression Recursive Feature Elimination is tested on real datasets in comparison with other typical feature selection methods,such as PCA,stepwise,LASSO,and Support Vector Regression Recursive Feature Elimination.In addition,we also select features randomly as a benchmark to demonstrate the rationality of our method.Our experimental results indicate that Distance Weighted Support Vector Regression Recursive Feature Elimination leads to better performance than other feature selection methods on several UCI benchmark datasets.