Ensemble Kalman Filter Based Optimization On Parameters Of Support Vector Machine

Support Vector Machine(SVM) is one of the general machine learning algorithms. The optimal SVM model is constructed by tuning various parameters, including SVM hyperparameters, feature weights, Lagrange multipliers for SVM training, and model thresholds for application. Handling massive data is more challenging for SVM. To improve efficiency and performance, this study proposes the application of Ensemble Kalman Filter(EnKF) for tuning parameters and features. EnKF is mainly used for data assimilation in geoscience. According to SVM algorithm and parameter optimization, this study proposes the Cascade Acceleration method to decrease the overhead of SVM training. Besides, gravitational-wave noise artifact analysis is as the main use case. This study also proposes a hierarchical model and a ROC patch algorithm to improve the identification of noise artifacts in gravitational-wave data. The main contributions are as follows:(1) An EnKF-based approach is proposed for SVM hyperparamter optimization. To overcome algorithm defects, several methods are proposed: multiple ensembles are used to avoid local optima; ensemble evoluation is defined to enpand the search; An EnKF based ensemble fusion algorithm is proposed to improve the representativeness of ensembles; A householder-based UR decomposition method is proposed to reduce the overhead of EnKF analysis calculation. The EnKF-based scheme is constructed based on the above methods. Detailed test results show that the EnKF-based approach can achieve better optimization effects comparied with three up-to-date Baysian algorithms within a given cost.(2) An EnKF-based approach is proposed for both feature selection and feature weighting. The hybrid optimization strategy is aimed to handle the high dimensional search space. The filter method is used for dimension reduction and guide of ensemble creation. Two-stage ensemble evolution is designed to improve the search efficiency. Moreover, the EnKF-based approach can optimize hyper-parameters and features on concurrency. Detailed test results show that EnKF-based approach can largely reduce features and achieve great improvements on some cases within a given cost.(3) A cascade acceleration method(CAM) is proposed to reduce the overhead of SVM training during the parameter optimization. The correlation is constructed between tasks according to the low dimension principle. The model result is picked as an initial condition to speed up the following SVM training through reducing the selection of working set. The test results show that CAM can decrease the SVM training overhead from 29.6% to 84.5% for grid search and from 21.9% to 62.7% in average for other optimization methods.(4) A hiearachical model is proposed to handle the gravitational-wave noise artifact analysis, which is a cost-sensitive classification problem. An unbalanced tree-structured SVM model is designed for classifying noise artifacts level by level. Besides, the ROC patch algorithm is proposed for threshold optimization and performance visualization. Detailed test results show that the hierarchical model can achieve 10%(relative) improvements on identification of noise artifacts.