Efficient Multi-objective Neural Architecture Search Based On Parallel Evolutionary Algorithm

In the past ten years,deep learning has achieved good results in various application fields,such as image classification,speech recognition,natural language processing,etc.And it mainly relies on a well-designed neural network architecture.Nowadays,most of the current network architectures are designed and developed by human experts,which is time-consuming and error-prone.Therefore,neural architecture search(NAS)technology was brought into existence.Compared with traditional automated machine learning technology,NAS is no longer limited to the optimization of hyperparameter search in deep learning,for example,the learning rate,but extends the search to more critical parameters,such as the types and sizes of convolution kernels.In the process of neural architecture search,the complete training and evaluation of a deep network architecture is very time-consuming,often taking several hours to several days.In addition,selecting a suitable architecture effectively in a large search space is also a key to neural architecture search.If the accuracy of the network is used as the unique target of the search,the network structures will be very large and complex,which are not inappropriate in resourceconstrained mobile devices,such as common mobile phones.In response to the problems mentioned above,this paper proposes an efficient multi-objective neural architecture search based on parallel evolutionary algorithm.In the search process,both the network accuracy and computational complexity are considered by adopting multiobjective evolutionary algorithm.Meanwhile,parallel strategies and surrogate model are used to make full use of hardware resources and historical evaluation data to speed up the evaluation and search process of neural architecture,and finally realize efficient multi-objective neural architecture search.The main research contents of this paper are summarized as follows:(1)An efficient multi-objective neural architecture search based on parallel evolutionary algorithm is proposed.Firstly,make full use of the trade-off ability of multi-objective evolutionary algorithm in multiple conflicting targets and the distribution of individuals to improve the efficiency of the algorithm to explore the search space.It can obtain a set of different network architectures distributed in the front to satisfy the deployment requirements of hardware devices with different computing capabilities in one run.Moreover,through synchronous and asynchronous parallel strategies,the evaluation of the network architecture is distributed to different child nodes for training in individuals and populations respectively.Parallel strategies are obvious to improve the evaluation speed of the network architecture and improve the efficiency of the algorithm.(2)On the base of the synchronous parallel strategy,the random forest surrogate model based on ensemble learning is constructed,and an efficient parallel multi-objective neural architecture search based on surrogate model is further proposed.In the design of the algorithm,the surrogate model is first constructed based on a small amount of initial dataset.In the subsequent search,performing K-Means clustering and sampling in each generation of individuals(i.e.,the network architectures),and using new dataset from actual evaluation to update the surrogate model.The cheap evaluation of the network architecture through the surrogate model accelerates the speed of neural architecture search further.(3)The proposed algorithm is evaluated on CIFAR-10 dataset,and compared with traditional artificially designed neural networks and representative excellent NAS methods to verify the precision and efficiency of the algorithm proposed in this article on the image classification problem.The experimental results on the CIFAR-10 dataset show that the efficient multi-objective neural architecture search based on parallel evolutionary algorithm designed in this paper significantly improves the search efficiency of the algorithm.Compared with other algorithms,it also has better competitiveness in network precision and params.In addition,combining the neural architecture search method proposed in this paper with an online surrogate model mechanism,the algorithm efficiency has been improved further.