Research On Sparse Learning Based Multiple Indefinite Kernel Learning For Feature Selection Algorithms

Feature selection refers to the process of selecting some of the most effective features from the original features sets to reduce the dimensions of the datasets.It can reduce the complexity of the model and the risk of overfitting.In recent years,there are many researches on feature selection algorithms.Multiple kernel learning for feature selection?MKL-FS?utilizes kernels to explore complex properties of features and performs better in linear and nonlinear feature selection algorithms.However,MKL-FS has two limitations:?1?the choice of kernel function is not rich enough;?2?the solution of feature selection is not sparse enough.On the one hand,MKL-FS requires the kernel function to satisfy the positive definite condition,which constrains the rich expression ability of the kernel function to some extent.Recent research shows that indefinite kernels can better characterize the relationship between data and achieve better results than positive definite kernels in many practical applications.However,due to the non-convexity of indefinite kernels,existing MKL-FS methods are usually inapplicable and the corresponding research is also relatively little.On the other hand,previous MKL-FS methods usually use the l₁-norm to obtain sparse kernel combination coefficients.However,the l₁-norm,as a convex approximation of l₀-norm,sometimes cannot attain the desired solution of the l₀-norm regularizer problem and may lead to prediction accuracy loss.Since the optimization problem of the l₀-norm is NP-hard,many linear feature selection methods use various non-convex approximations of l₀-norm to replace l₀-norm,and have achieved good results.However,non-convex approximations are rarely used in nonlinear models currently.This paper will improve the MKL-FS methods from these two aspects and the work are summarized as follows:1)Since MKL-FS methods are limited to using positive definite kernels,this paper proposes a novel l₁-norm based multiple indefinite kernel learning for feature selection?l₁-MIK? method based on the primal framework of indefinite kernel support vector machine?IKSVM?,which applies an indefinite base kernel for each feature and then exerts an l₁-norm constraint on kernel combination coefficients to select features automatically.In order to solve the non-convex optimization model,a two-stage algorithm is proposed to optimize the IKSVM coefficients and the kernel combination coefficients alternatively.In the algorithm,the non-convex problem of the IKSVM is reconstructed into a difference of convex functions?DC? programming,and solved by DC algorithms.The optimization problem of the kernel combination coefficient is solved by the projected gradient method.In order to further extend to large-scale problems,this paper uses a leverage score method to sample large-scale datasets,and extends the l₁-MIK method to multi-class classification scenarios.Finally,the effectiveness of the proposed algorithm is verified on the actual datasets.2)Since MKL-FS methods are limited to using l₁-norm for feature selection,this paper further proposes a novel l₀-norm based multiple indefinite kernel?l₀-MIK?method for feature selection with non-convex approximations constraint on kernel combination coefficients to select features automatically.l₀-MIK is based on the primal framework of IKSVM and is also solved by a two-stage algorithm,in which the non-convex problem of the IKSVM and the non-convex problem of the l₀-norm are reconstructed into DC programming,and solved by DC algorithms respectively.Extensive experiments show that l₀-MIK is superior to the existing MKL-FS methods and l₀-norm based linear feature selection algorithms in terms of classification classification accuracy and sparsity of features.