| The development of machine learning models is extremely time-consuming and laborious.In order to reduce the cost of model development,the researchers proposed the concept of Automated Machine Learning(AutoML),which is intended to automatically complete a series of operations such as algorithm selection,hyperparametric configuration and neural architecture design without human participation,so as to realize the automatic and efficient development of high-precision machine learning models.However,there are a large number of feasible schemes in each step of model development.In order to quickly select the most suitable solution for a given task from them,AutoML algorithms that are as intelligent as domain experts are needed as support.The key to intelligent decisionmaking of domain experts is”knowledge”.They have a lot of relevant knowledge and can quickly extract effective knowledge to help them make fast and high-quality decisions.However,the existing AutoML algorithms mainly focus on designing search skills,failing to use or make full use of relevant knowledge to improve algorithm performance.In order to further improve the efficiency of automatic development of machine learning models,this paper proposes to inject knowledge into AutoML techniques,proposing efficient knowledge-driven AutoML algorithms for the three key problems of the AutoML:hyperparameter optimization problem,combined algorithm selection and hyperparameter optimization problem,and neural architecture search problem.The algorithms proposed in this paper can make full and reasonable use of known information,including historical evaluation information,meta information,domain knowledge and domain design experience,to extract knowledge that is helpful for decision-making or select useful knowledge,and thus effectively improve the efficiency and quality of the automatic model generation,which is more practical than the existing algorithms.In addition,in order to promote the application of the above three AutoML techniques,this paper also explores the application scenarios of these techniques in depth,refine and analyze the use mode and application value of these techniques.The main research contents of this paper are summarized as follows.1.In this paper,the Hyperparameter Optimization problem is studied.Hyperparameter Optimization(HPO)aims to find the optimal hyperparameter configuration for the machine learning algorithm that is sensitive to its hyperparameters.This problem is usually a black-box optimization problem.To find an excellent hyperparameter configuration under a small budget,it is needed to make full use of historical evaluation information to extract effective knowledge to better understand the characteristics of the target task,so as to improve the possibility of obtaining an excellent solution.Therefore,this paper designs two effective hyperparameter configuration selection methods.They can quickly extract effective knowledge that is helpful for decision-making and learn the importance of each hyperparameter from a small amount of historical evaluation information,intelligently infer the optimal configuration from two aspects:knowledge utilization and search space pruning.Two methods are suitable for different task scenarios and complement each other.This paper proposes to effectively combine two methods to improve the universality and effectiveness of the proposed algorithm.This paper uses UCI datasets and image classification datasets to conduct experiments on a variety of HPO tasks and verifies the rationality and effectiveness of the proposed algorithm:Under the condition of a small amount of computing resources,the algorithm proposed in this paper that is based on knowledge analysis and pruning is more efficient compared to the traditional HPO algorithms.2.This paper studies the problem of Combined Algorithm Selection and Hyperparameter Optimization problem.Combined Algorithm Selection and Hyperparameter Optimization(CASH)problem aims to select the best algorithm from the existing machine learning algorithms,and find the best hyperparameter configuration for it at the same time.Because there are many options of the machine learning algorithm and the hyperparameters of different algorithms vary greatly,the CASH problem has a huge search space and high search difficulty.A quick understanding of the characteristics of each algorithm in the search space is helpful to improve the quality of decision-making and reduce search difficulty,but this needs effective knowledge as support.Therefore,this paper proposes a solution based on meta learning,which makes full use of the experimental results reported in the relevant research papers to extract effective meta-knowledge,and thus achieve a quick and reasonable selection of the well-performed algorithm,and then selects an appropriate HPO method to quickly solve this problem.The method proposed in this paper can greatly reduce the cost of meta information acquisition and reduce the search space,so it is highly efficient.This paper conducts experiments in the background of classification algorithm selection and hyperparameter optimization,uses a large number of classification datasets to verify the efficiency of the proposed method:Compared with the classical non-meta-learning methods,the method proposed in this paper can provide better results in a short time,achieving better performance with the help of meta-knowledge.3.In this paper,the problem of Neural Architecture Search is studied.Neural Architecture Search(NAS)aims to automatically design a new and excellent neural network model.The key to solve this problem is to design the high-quality search space and the efficient architecture search strategy.The existing algorithms mainly focus on convolutional neural networks.Their search space construction methods are not suitable for non-convolutional machine learning models,having poor universality.In order to successfully extend NAS technique to other types of models,and considering that the design experience and knowledge in the domain can help to quickly grasp the design rules of the models in this domain,this paper proposes a NAS algorithm based on domain knowledge.This algorithm can make full use of the design experience of functional modules in the domain to quickly construct a high-quality architectural search space suitable for a specific domain,and quickly understand the characteristics of each module through domain knowledge,so as to quickly build a high-quality model architecture.This paper carries out experiments on the trajectory dataset in the background of the trajectory prediction,and verifies the effectiveness of the proposed method:With the help of domain experience and knowledge,the method proposed in this paper exceeds the existing NAS algorithm and can quickly design better trajectory prediction models.4.This paper studies the application problem of AutoML techniques.In recent years,in order to tackle the above three problems,researchers have proposed many AutoML algorithms,but the application scenarios of these three types of techniques are not very clear,which brings challenges to their further promotion.To make more users use these techniques,this paper deeply explores the application scenarios of these techniques,and proposes two application scenarios:interactive application scenarios for the existing research directions,and integrated application scenarios for the new research directions.The former integrates AutoML techniques into an interactive system for users to use,while the latter utilizes the AutoML technique and integrates multi-domain design experience to design new models.This paper details the use of three types of AutoML techniques in two scenarios,and makes demonstration or experimental analysis on the classification algorithm HPO task and the trajectory prediction task respectively.The demonstration results and experimental results verify the importance of the three types of AutoML techniques analyzed in this paper:They can not only facilitate the development of conventional models,but also help the research of new machine learning models,their practical values are high. |
