Studies On The Black-box Optimization Methods

Black-box optimization is an important type of optimization problems,which is commonly used to tune hyperparameter in automated machine learning and comput-er experiments.The objective function to optimize is unknown,so it is regarded as a black-box process,from which it got its name.Nowadays,there are many black-box optimization methods,which can be divided into two categories:(i)Bayesian opti-mization algorithm based on surrogate model.(ii)algorithms that do not depend on the model,which mainly include sampling methods such as grid search and random search,and intelligent optimization algorithms such as genetic algorithm and parti-cle swarm algorithm.The most widely used Bayesian optimization algorithm is the GP-EI algorithm,which builds a surrogate model(Gaussian process)based on the initial test points,then selects the point that maximizes the expected improvement function as the new test point,and then iteratively updates.The most significant,shortcoming of the GP-EI algorithm is that,each iteration only adds one test point,which reduces the resource utilization rate,cannot use efficient parallel computing,and requires a large time cost.Many scholars have proposed various improvement methods,mainly the multi-point El algorithm,multi-model EI algorithm,etc.These methods generate multiple update points which have a larger EI value at the same time during each update process.In practical applications,parallel algorithms are used to test at multiple points at the same time,thus speeding up the optimization process.However,optimizing multiple points for the EI function at the same time also brings about the problems of computational complexity and test point aggregation.The accelerated EGO algorithm proposes a new multi-point generation method,which regards the El function as a similar probability distribution function,and uses the importance re-sampling method to directly extract points with larger EI values from the sample pool.The advantage of this is that the computational complexity can be greatly reduced,and the space filling of the sample pool can be used to avoid the accumulation of test points.But it is worth noting that the sample quality obtained by the importance re-sampling method largely depends on the selection of the sample pool.However,there is no systematic and scientific discussion on the sample pool in the existing literature.This paper systematically compares the performance of the accelerated EGO algorithm under different sample pool,and finds that LA-EGO(Accelerated EGO algorithm based on Lattice design)and UA-EGO(Accelerated EGO algorithm based on uniform design)has better optimization capabilities in low-dimensional space,and under the same number of trials and iterations,the convergence speed is faster and the algorithm runs less time.In contrast,SA-EGO(Accelerated EGO algorithm based on Sobol sequence)performs better in the high-dimensional test area,and usually finds a result closer to the global optimum in a relatively short time.In addition,in order to examine the effects of the above-mentioned optimization algorithms in hyperparameter tuning,this paper takes two machine learning models,support vector machines(SVM)and extreme gradient booster models(XGBoost),as examples,and uses multiple optimization methods on four data sets.The results are compared.This paper mainly chooses grid search,random search,GP-EI,SMAC,TPE,SA-EGO,UA-EGO and LA-EGO.Simulation experiment results show that although grid search and random search have the fastest search speed,the opti-mization results are not as good as other algorithms.Compared with the GP-EI algorithm,the SA-EGO,UA-EGO and LA-EGO methods can search for the best results faster.On the SVM,the GP-EI algorithm performs best among the three Bayesian algorithms.In the case of the same number of trials and iterations,among the accelerated EGO algorithms under the three sampling methods,the LA-EGO method performs better.On the XGBoost,the GP-EI algorithm does not perform as well as the other two Bayesian optimization algorithms.However,SA-EGO and LA-EGO are still a good improvement to the GP-EI algorithm,and the SA-EGO method can obtain better test results.Generally speaking,in the low-dimensional test space,the accelerated EGO algorithm based on Lattice design can obtain better test results,while in the high-dimensional test space,the accelerated EGO algorithm based on Sobol sequence performs better.