Well-Generalizable Linear Dimensionality Reduction Algorithms In Classification And Regression Tasks

The curse of dimensionality leads to the weakness of generalization ability for various machine learning models.Linear dimensionality reduction has low computational cost and strong geometric interpretations.Therefore,for different models,this dissertation studies the corresponding well-generalizable linear dimensionality reduction algorithms in classification and regression tasks.For k-nearest neighbor classifiers,supervised discriminative sparse principal component analysis with adaptive neighbors(SDSPCAAN)was proposed,which integrates supervised discriminant sparse principal component analysis and projected clustering with adaptive neighbors to simultaneously preserve both global and local structure information for high discriminant feature extraction.Experimental results demonstrated that,for k-nearest neighbor classifiers in all subspace dimensionalities,SDSPCAAN almost always achieved the highest balanced classification accuracies and strong parameter robustness.For penalized least squares classifiers(PLSCs),joint supervised discriminative sparse principal component and discriminant analysis with adaptive neighbors(JSDSPCDAN)was proposed,which integrates SDSPCAAN and PLSC,and utilizes the Manopt toolbox to improve the joint optimization strategy for end-to-end discriminant feature extraction and classification.Experimental results demonstrated that,for PLSCs in all subspace dimensionalities,JSDSPCDAN almost always achieved the highest balanced classification accuracies and strong parameter robustness.Additional experiments demonstrated that for k-nearest neighbor classifiers and PLSCs in all subspace dimensionalities,JSDSPCDAN,which integrates SDSPCAAN and PLSC,always performed as well as,if not better than,SDSPCAAN in generalization ability with increased computational cost.For PLSCs,multi-view broad learning system(Mv BLS)was proposed,which extends the PLSC-based broad learning systems to multi-view scenarios,and utilizes multiple differently random-initialized linear sparse auto-encoders to reduce the feature dimensionality on each view for multiple times,so as to preserve the incremental learning characteristic,avoid the interfere among views,select and extract multi-view diversified features.Experimental results demonstrated that,compared with other single-view and multi-view algorithms,Mv BLS could effectively extract the complementary information about oculomotor decisions of local field potentials and action potentials collected from the rhesus monkeys’ supplementary eye field in the medial frontal cortex,significantly improved the decoding accuracy and efficiency.For Takagi-Sugeno-Kang(TSK)regression fuzzy systems,Powerball Ada Belief gradient descent algorithm was proposed,which adds a gradient module rescaling to existing gradient descent algorithm,relieves the gradient vanishing and exploding problems brought by high-dimensional input and outliers to accelerate and stabilize the gradient descent optimization process.Integrating the proposed algorithm with strategies like regularization,leads to TSK fuzzy systems with strong generalization.Experimental results demonstrated that,for all numbers of rules,the proposed algorithm always achieved the lowest root mean squared errors and strong parameter robustness.For TSK regression fuzzy systems,consistent dimensionality reduction(CDR)and two variants.CDR automatically identifies a shared informative input subspace for the rule antecedents and consequents via gradient backpropagation,and thus performs wellgeneralizable end-to-end consistent feature extraction.Integrating CDR and the above gradient descent algorithm,with grid partition initialized shared membership functions,or fuzzy c-means clustering initialized independent membership functions,leads to two different TSK regression fuzzy systems.Integrating CDR with adaptive rule pruning,or adaptive group LASSO,leads to two variants.Experimental results demonstrated that,for all numbers of rules,CDR almost always achieved the lowest root mean squared errors,high computational efficiency,and strong parameter robustness.Two variants further achieved stronger generalization ability with slightly increased computational cost.In summary,for the three machine learning models(k-nearest neighbor classifiers,PLSC,and TSK regression fuzzy system)suffering from the curse of dimensionality,this dissertation made a systematic study on linear dimensionality reduction algorithms with strong generalization ability in the corresponding high-dimensional classification and regression tasks.The proposed SDSPCAAN that simultaneously preserves both global and local structure information for k-nearest neighbor classifiers,JSDSPCDAN that performs end-to-end discriminant feature extraction for PLSCs,Mv BLS that selects and extracts multi-view diversified features for PLSCs,and CDR-FCM-RDp A that performs end-to-end consistent feature extraction for TSK regression fuzzy systems,all achieved strong generalization ability.