Evolutionary Computation For Deep Neural Network Architecture Search

Deep neural networks have achieved remarkable achievements in a variety of computervision-related tasks,such as image classification and object detection.One of the main driving forces behind this success is the network architecture,and it is also the basis for the entire model to be efficiently trained and applied.Most of the existing high-performance models are manually designed by human experts.However,designing a successful hand-crafted network architecture depends on extensive expertise in both deep learning and the related application area,which is inconvenient to many practitioners.With the advent of the information age,the forms and sources of data are increasingly rich.It is now well recognized that designing task-specific neural network architectures for different scenarios is one of the most challenging of the entire DNN model development process.Neural architecture search poses the design of network architectures as an optimization problem,which algorithmically alters the network architectural components.As a result,novel network architectures can be searched that exhibit improved performance metrics on given datasets.Although the research on NAS methods has made remarkable progress,several grand challenges remain,in particular,the prohibitive computational resources required for performing exploratory neural architecture search.Based on the correlated theories and mechanisms of evolutionary computation,this dissertation focuses on improving the computational efficiency and performance improvement of evolutionary neural network architecture search algorithms and proposing several novel search algorithms.The contributions of this Ph.D.thesis include:(1)A fast evolutionary neural architecture search framework is proposed,which is well suited for implementation on devices with limited computation resources.The computational costs of fitness evaluations are dramatically reduced by two related strategies,sampled training of the parent individuals and node inheritance for the weights of the offspring individuals.To further improve the expression ability in evolving large neural networks,the multi-scale feature reconstruction convolution operation is encoded into search space.Our experimental results show the proposed algorithm is not only computationally much more efficient,but also highly competitive in learning performance.(2)An evolutionary one-shot neural architecture search framework is proposed,which can reduce,to a certain degree,the negative impact of multi-model forgetting in the one-shot model and directly generates network architectures on target tasks without any proxy metrics.This objective has been achieved by proposing a new node representation scheme for network architectures,together with corresponding crossover and mutation operators.To further improve the accuracy and efficiency in evolving large network architectures,we encode the Py Conv operations into search space,which mixes multiple convolutional kernels in a single operation to utilize different kernel sizes.Experiments on four benchmark classification tasks,including CIFAR,CINIC10,and Image Net,have demonstrated the effectiveness of the proposed method.(3)A fast evolutionary neural architecture search framework based on one-stage Retina Net for the object detection task is proposed.The computational burden of fitness evaluations is significantly reduced by two related strategies,individual parameter mapping and individual distributed training.The individual parameter mapping strategy can map the parameters of the pre-trained teacher detector to a new individual(candidate detector)as the initial parameters with negligible cost,thereby avoiding the Image Net pre-training of each individual's backbone during the evolutionary search.The individual distributed training strategy can dramatically speed up the fitness evaluation with limited computation resources.To further improve the capability of generating multi-scale features,the Fa PN module is incorporated into the Retina Net framework to replace the FPN module of the feature fusion neck part.Our experimental results demonstrate that the proposed method can not only accelerate the evolutionary architecture search but also achieve promising performance compared with the state-of-the-art NAS methods.(4)Finally,based on the analysis of biomedical image classification problems and textile defect detection problems,we apply the proposed algorithms to design task-specific network architectures in different data scenarios.The experimental results show that the proposed methods outperform the existing classification and detection methods.