Neural Architecture Search Algorithm Optimization Based On Differentiable

Deep learning has shown excellent performance in artificial intelligence fields,such as computer vision,natural language processing,robotics and so on,because it provides powerful automatic feature extraction and representation capabilities for unstructured data.For different practical application scenarios,many high-performance neural network structures have been manually designed,but manual design of network structures often requires rich expertise in the target domain and extremely high labor costs.In order to solve the above problems,automatic design of neural network structure has been widely concerned by academia and industry.Neural Architecture Search is currently the mainstream method of automated network structure design.Among them,the differentiable neural architecture search algorithm relaxes the discrete architecture search space into a continuous one,so as to use the gradient to optimize the architecture parameters to search the network structure,thus greatly reducing the consumption of computing resources and promoting the actual deployment of the NAS algorithm with application.However,the continuous relaxation of search space and differentiable search strategy not only efficiently complete the network structure search,but also lead to a series of inherent defects of differentiable NAS algorithm.Therefore,this thesis aims to further optimize the differentiable neural architecture search algorithm,make up for its own inherent defects and make the algorithm have hardware awareness according to the actual application requirements,and improve the search performance and structure generalization ability of the differentiable NAS algorithm.Firstly,this thesis presents multi-path restricted differentiable architecture search.Aiming at the inherent depth gap and search collapse problems of differentiable neural architecture search,we propose a restricted connectivity algorithm and optimize the parameter mapping method and loss function of the algorithm.The restricted connectivity algorithm deepens the depth of architecture building cells,making it more suitable for deep networks,thereby reducing the number of stacks of the cells for evaluation network,and making up for the depth gap.The designed 0-1 coefficient loss and normalized cooperative parameter mapping function solve the search collapse problem simply and efficiently,enabling the algorithm to perform more accurate searches under limited computing resources.Meanwhile,the proposed method also fill the gap in the application of differentiable NAS to multi-path search space.The designed multi-path search space adapted to the differentiable search strategy enables the architecture building cells to extract and fuse features of different scales,which improves the feature extraction and representation capabilities of the network.The multi-path restricted differentiable architecture search algorithm achieves state-of-the-art performance on benchmark datasets with the least number of parameters(only 2.5M)and the searched structures exhibit strong generalization ability.Next,differentiable NAS algorithm is difficult to make the search strategy have hardware awareness due to its complex search space representation.However,the popularity of mobile terminals and the expansion of artificial intelligence application scenarios urgently require the network structure to have the characteristics of low inference latency in the target hardware device.Therefore,this thesis proposes a hardware latency-aware differentiable neural architecture search.First,we construct a latency prediction proxy dataset of the target hardware device,and then use the soft attention mechanism and Mish activation function to build a hardware latency prediction network,so as to accurately predict the delay of the latency structure in the complex search space.Finally,the hardware delay constraint is embedded in the differentiable neural architecture search.The proposed method can be transferred to different hardware devices with high efficiency and low cost,and a high-performance network structure with both accuracy and inference delay can be obtained by searching,which meets the practical application requirements of differentiable NAS.This thesis studies and realizes the optimization of the differentiable neural network architecture search algorithm.The proposed method realizes the accurate network structure search with low resource consumption,and meets the high requirements of complex artificial intelligence tasks on network performance.Furthermore,it has important theoretical significance and practical value for the deployment of neural networks on mobile terminals and edge computing devices.