Algorithm And System Research On Efficient Deep Ensemble Architecture Search

In recent years,deep learning methods represented by deep neural networks have achieved great success in image,text,speech,video and other tasks,where the idea of"depth"is considered the key to deep learning.However,in practical applications,deep neural networks face problems such as high computational overhead,high training set size requirements,and poor support for tabular data sets.In order to solve the above problems,the academic community began to try to integrate the idea of"deep"and the advantages of integrated learning methods,and research to propose a deep ensemble learning model represented by deep forest.Similar to DNN architecture design,designing a deep ensemble architecture with good performance for different input datasets and task scenarios requires a lot of expert knowledge and experience,and it is time-consuming to repeatedly try various combinations of hyperparameters such as different base learner types and base learner integration methods.In recent years,Neural Architecture Search（NAS）technology has attracted a lot of attention from industry and academia.Inspired by NAS,this paper will investigate the implementation of efficient deep ensemble architecture search methods to automate the search for deep ensemble architectures with excellent performance for specific datasets and task scenarios,thereby greatly reducing the design threshold of deep ensemble learning models and improving the efficiency of deep ensemble learning modeling.However,deep ensemble architecture search needs to solve a series of challenges such as search space design,architecture search method design,and architecture search optimization acceleration.To this end,this paper firstly researches and designs an effective deep ensemble architecture search space,and proposes an evolutionary algorithm-based deep ensemble architecture search method and an agent model-based incremental deep ensemble architecture search method.Based on the above research,an automated deep ensemble learning system is further designed and implemented,and the architecture search process is accelerated by Ray,a distributed computing framework.The main research points and contributions of this paper include:（1）Study and design two effective deep ensemble architecture search spaces,including fully parallel and directed acyclic graph-based search spaces,and give formal definitions of these two search spaces.By analyzing the characteristics of existing deep ensemble learning models,the study designs two deep ensemble architecture search spaces,called fully parallel search space(_CP)and directed acyclic graph search space(_DAG),respectively,and the proposed two search spaces change the situation that the original deep ensemble learning methods are dominated by deep forests,enabling various automated search methods to be applied to deep ensemble learning.（2）The research proposes EPEAAS,an evolutionary algorithm-based deep ensemble architecture search method.based on the definition of the search space,the method of deep ensemble architecture search using the optimal individual-preserving evolutionary algorithm is proposed.In addition,and by abstracting the above-mentioned evolutionary algorithm-based architecture search method,a general deep ensemble architecture search algorithm framework based on evolutionary algorithm is proposed.The experimental results show that EPEAAS has a more obvious performance improvement on classification and regression tasks compared with ensemble learning methods,neural networks,and deep forest methods.（3）PMPAS,a progressive deep ensemble architecture search method based on proxy model,is proposed to analyze the problems of EPEAAS,such as high computational overhead and low search efficiency,and propose the basic idea of building deep ensemble learning architectures from simple to complex,and adopt the proxy model to predict the architecture performance,so as to reduce the architecture evaluation overhead and improve the architecture search efficiency.The experimental results show that PMPAS obtains large performance improvements on classification and regression tasks compared to ensemble learning methods,neural networks,deep forest methods,and automated machine learning methods auto-sklearn.In addition the comparative experimental results of EPEAAS and PMPAS show that both search methods proposed in this study are effective in searching excellent deep ensemble architectures with their respective applicability scenarios.（4）Based on the studies of EPEAAS and PMPAS,further design studies are conducted to implement Auto-DEL,a system that supports the automated construction of deep ensemble learning architectures,and accelerate Auto-DEL by Ray,a distributed computing framework,to enhance the scalability of the system,.In addition,and enhance the ease of use of the system by providing a simple and easy-to-use API.