Neural Architecture Search Based On Evolutionary Multi-Objective Optimization

In recent years,the research related to deep learning has achieved great success,and the application of deep learning algorithms has become more and more important.However,the design and construction of neural network models rely on a large number of professionals and consume a lot of time.Moreover,as the tasks become intricate,the structure of the neural architecture becomes more and more complex,and the problems of high dependence on professional knowledge and high construction costs have become more and more obvious.Therefore,the concept of Automated Machine Learning(AutoML)was proposed,many researchers have tried to use algorithms to automatically build deep learning models.As an important part of AutoML,neural network structure search(NAS)has also been greatly developed.Among them,the neural architecture search based on evolutionary algorithm has attracted the attention of researchers because of its outstanding performance.However,the existing methods based on evolutionary algorithms have the problem of inflexible representation of the neural architecture,and the huge search space causes the search algorithm to be very inefficient.The work of this thesis is mainly to propose a neural architecture encoding strategy and a search strategy that can effectively solve the above two problems.(1)In order to solve the problem of inflexible representation of neural architecture,this thesis proposes the neural architecture encoding strategy ACEncoding based on action commands.In evolutionary algorithms,a good encoding strategy can greatly improve the performance of the search algorithm.ACEncoding is a modular encoding strategy that draws on protein expression.It summarizes the construction of the neural network into 2 basic actions and 5 combined actions derived from the basic actions.Each specific action is represented by a three-digit symbol,the first digit represents the specific action,and the second and third digits represent the parameters required to perform the specific action.Compared with other encoding strategies,the ACEncoding encoding strategy has a more compact search space and can use shorter codes to represent more complex neural architectures.Moreover,due to the modularity of the ACEncoding encoding strategy,the strategy can also be extended according to specific tasks.In addition,in order to reduce the time-consuming evaluation of neural architectures and improve search efficiency,this thesis also proposes a surrogate model Seq2 Rank for evaluating the performance of neural architectures.Seq2Rank is a surrogate model that can receive variable length input.Compared with traditional neural architecture evaluation strategies,Seq2Rank can increase the speed while ensuring that the performance is almost not reduced.(2)In order to solve the problem of low efficiency of search algorithm caused by huge search space,this thesis proposes Transfer NAS,a knowledge transfer neural architecture search algorithm based on multi-objective evolutionary algorithm.The Transfer NAS algorithm uses pre-learning strategies to learn knowledge on simple tasks and build a knowledge map.The knowledge map will use knowledge-assisted search algorithms learned in simple tasks to quickly locate areas in the huge search space that may have better solutions and exclude those meaningless search areas,thereby guiding the search algorithm to quickly explore in the huge search space.At the same time,in order to weaken the difference between simple tasks and complex tasks,the knowledge map will dynamically update the learned knowledge according to the search results during each generation of the evolutionary algorithm to ensure the effectiveness of the knowledge map.Experiments show that,compared with the existing neural network structure search algorithm,Transfer NAS has a faster convergence speed,and can search the same performance neural network structure that the existing algorithm needs to be searched in the middle or later stages of the search.