Research On Hyperparameter Optimization For Deep Learning Models

Deep learning models have been extensively applied in various fields,and the performance of algorithm model is decided by its own structure.It can be rather difficult to find a set of optimum hyperparameters to determine the network structure,which presents as a major obstacle for the application of deep learning model.While the hyperparameters cannot be obtained through the model training,it must be set no later than the model runs.At present stage,most methods are operated mostly rely on previous experiences and manual selection,which is inefficient.When encountered with huge volume of data or complex structures,it can be hard to manually adjust parameters.Therefore,the way of realizing efficiently automatic parameter adjustment,from the perspective of model itself,serves as the key of hyperparameter optimization.Conversely,the optimization of hyperparameter also means significantly for the effective application of model algorithm.Therefore,this paper lays its focus on the hyperparameter optimization of deep learning model.Firstly,the application effect of six optimization algorithms on BP neural network and LSTM is respectively studied.These methods specifically involve grid search,random search,Bayesian optimization,Talos optimization,hyperband and Optuna framework.Secondly,the finest hyperparameter optimization method for BP and LSTM is respectively illustrated by comparing the time efficiency of optimization with the accuracy of the final model.More specifically,the hyperparameter optimization method for BP neural network is hyperband,and Optuna is prepared for LSTM,which provides optimization experiences for more widely used models.A hypothesis is hereby proposed in the end of this paper: whether the varied optimal algorithm is also corresponded to varied types of neural networks(feedforward neural network and cyclic neural network).Additionally,the author has also demonstrated that different hyperparameter optimization methods is corresponded to various deep learning issues.To cope with the simple problem encountered in BP neural network,the optimum hyperparameter optimization method is hyperband,and Optuna serves as the optimum one for LSTM.