Study On Neural Network Architecture Search With Evolutionary Strategy

Artificial neural networks(ANNs)is a kind of information processing mode in the field of data-driven artificial intelligence.It has been successfully applied in many fields,such as image recognition,speech processing and machine translation,by simulating the signal processing of biological neural networks.However,the performance of neural network is easily affected by network topology,connection weights and other factors.At present,the architecture of the popular neural network has become more and more deep and complex.And hand-crafting deep neural network architectures,however,is a laborious task.In order to achieve the best performance on specific data sets,a huge number of parameters need to be adjusted for the artificially designed deep neural network.Therefore,when building an optimal architecture for different data,designers need to have a high level of expertise and experience and spend a lot of time and computing resources.In view of this,the automatic design of artificial neural network architecture becomes an urgent problem to be solved.Artificial neural networks can be thought of as composed of a large number of layers and weights,and in the network layers and the weights of the arrangement of satisfying certain distribution rules.In order to enable the distribution rules to be used for the evolution of the neural network architecture,we proposed a new method of generating the optimal architecture,Evolutionary Strategy-based Architecture Evolution(ESAE).It consists of a bi-level representation scheme and two kinds of distribution learning strategies,evolutionary operator-based probability distribution learning strategy and weight distribution-based performance sorting learning strategy.Based on strategies and ESAE,this paper designs the Probability Distribution Learning Strategy-based Neural Architecture Search(PDNAS)and the Weight Distribution Performance Ranking-based Neural Architecture Search(WDNAS).The ESAE can not only search the best network architecture(such as layer type,layer parameter,etc.),but also adjust the weight of the network.The bi-level representation scheme encodes the layers and parameters of the network.The probability distribution learning strategy of evolutionary operator maintains dynamically various probability distributions,e.g.,genetic operators,parameters,which ensures the efficient convergence of the architecture searching process.The performance ranking learning strategy based on weight distribution completes the weight sampling and assignment of a given network,and accelerates the speed of network architecture searching by replacing the training of back propagation with the weight sampling and assignment.In order to verify the effectiveness of the PDNAS,we conducted experiments on the Fashion-MNIST and CIFAR-10.Compared with the existing state-of-the-art manual screwed neural network architectures,the ESAE searched from a trivial initial architecture with one single convolutional layer,and achieved the classification accuracy of 94.48% and 93.49% on Fashion-MNIST and CIFAR-10,respectively.Moreover,by adjusting the kernel size of convolutional and pooling operators and the convolutional channel numbers,PDNAS could be easily transferred to other fields.As an example,the evolved DNNs,trained with images,was applied on the recognition of human genomic signals and regions(GSRs)and reached quite agreeable accuracies of 95.48% for TIS and 85.86% for PAS(AATAAA),respectively.In addition,we find that the weight distribution of the convolutional neural network follows the normal distribution with the expectation of 0 and the standard deviation of 0.1,and the performance ranking of the network with similar complexity is consistent with the ranking of the forward propagation only by this normal distribution.Based on this finding,we replace the training process of neural network architecture in ESAE with the forward propagation based on weight sampling,and propose the Weight Distribution-based Neural Architecture Search(WDNAS),which accelerates the search process.On CIFAR-10,the architecture searched by our method achieves 93.76% test accuracy.