Research And Application Of Key Technologies Of Efficient Neural Architecture Search Based On Resource Awareness

In recent years,with the advancements in high-performance computing and big data technologies,deep learning has gained much popularity in Artificial Intelligence（AI）re-search community.Particularly,deep learning models,which are mainly based on Artifi-cial Neural Networks（ANNs）,simulate the human brain nervous system from the perspec-tives of structure,mechanism,and efficacy,and have revealed overwhelming performance in end-to-end learning from high-dimensional and complicated data.With the deepening of research,the architecture optimization and hyperparameter tuning of neural networks become extremely burdensome and strongly dependent on domain expertise.The Neural Architecture Search（NAS）aims to automate this process,to significantly reduce man-power consumption and dependence on expert knowledge.As an exploration of Auto-mated Machine Learning（Auto ML）technology in the deep neural network field,NAS has become an essential component of the Automated Deep Learning（Auto DL）pipeline.Resource-aware NAS searches the optimal architectures for a certain resource plat-form by simultaneously considering a series of objectives such as performance,inference efficiency,and resource consumption.Though abundant studies have been conducted,several limitations remain unsolved.First,the efficiency of the resource-aware NAS needs to be improved concerning search strategy,architecture evaluation strategy,and resource-aware form.Meanwhile,current resource-aware NAS algorithms rarely consider the char-acteristics of the deployment platforms when designing the predefined architecture,search space,and search strategy.The software-hardware co-search/co-design rarely involves non-von Neumann architecture platforms,which target future intelligent computing.This dissertation explores several key issues faced by resource-aware NAS research,including search space,search strategy,evaluation method,deployment performance,and proposes search methods/frameworks both for traditional architecture and processing-in-memory（PIM）architecture resource platforms.The specific contents and innovations are summa-rized as follows:1.Aiming at the inability of the differentiable NAS framework to tackle the non-differentiable search objectives（e.g.,energy,latency,or memory consumption）and the low efficiency of multi-objective NAS based on individual heuristic search strategies,this dissertation proposes the multi-objective NAS framework which is based on the differ-entiable hypernetwork and policy gradient optimization w.r.t the architecture parameters.This framework could provide high efficiency comparable to differentiable NAS while preserving the objectives compatibility of heuristic search strategy based multi-objective NAS.The proposed method exhibits promising results in both rendition and efficiency on image classification benchmark datasets and could serve as the search strategy in resource-aware NAS.2.The resource-aware NAS commonly relies on the high cost of online resource awareness,whereas the resource-aware surrogate model demands dramatic resource con-sumption in large-scale dataset collection and generally fails to achieve the generalization efficacy of large-scale search space with high prediction performance.This dissertation proposes the Latency/Accuracy prediction and the architecture binary-relation prediction surrogate models for a large-scale search space(Mobile Net V3,the scale of 10²⁵).Also,a multi-task learning-based binary-relation prediction/ranking surrogate model is proposed.The experiments imply that the binary-relation prediction model could provide satisfac-tory prediction performance based on a small number of labeled architecture points（e.g.,100）,by which to support the resource-aware multi-objective NAS.Meanwhile,the multi-task learning-based model achieves cross-task learning and provides a joint architecture binary-relation and ranking prediction performance that are superior to single-task learn-ing.3.The current NAS researches rarely consider the resource characteristics of PIM platforms（non-von Neumann）.This dissertation first empirically demonstrates the cor-relation among architecture,fixed-point quantification,and bit-level sparsity,that is,in a specific architecture search space,some architectures have a high tolerance to non-structured bit-level sparsity characteristics and are more efficient for deployment on PIM platforms.Further,this dissertation constructs a NAS framework oriented to the PIM re-source characteristics,which explore the bit-level sparsity-tolerant architectures.These architectures may significantly reduce the resource consumption of ADCs when deployed on PIM platforms,by which to appropriately compensate for the performance sacrifice caused by fixed-point quantization and bit-level sparsity from the way of architecture op-timization.4.The Graph Neural Network（GNN）computing based on the PIM platform usu-ally relies on the overall mapping of large-scale graph data（e.g.,adjacency matrices）on crossbars.Unfortunately,the current fabrication technology of crossbars is not yet ma-ture,and it is difficult to produce large-scale crossbars with high yields.For this con-tradiction,this dissertation proposes the spectral Graph Convolutional Network（GCN）deployment scheme based on sparse Laplace matrix reordering and diagonal block matrix multiplication,which effectively utilizes discrete small-scale crossbars,with considera-tion of efficiency and resource consumption（memristor number,crossbar area）.Based on this efficient deployment strategy and according to the resource characteristics of PIM-based large-scale graph data deployment,this dissertation further formulates the map-ping scheme as the 0-1 sequence decision problem and proposes the sparsity-aware based search method for dynamic mapping scheme,by which to meet the real constraints（e.g.,limited crossbar size,the peripheral circuits complexity,etc.）of the PIM deployment platforms.Experimentally,the searched mapping schemes achieve remarkable results on large-scale graph/matrix datasets.5.To facilitate the application of NAS,this dissertation introduces NAS technology to portrait/face segmentation/parsing tasks.An efficient encoder-decoder based search space is designed,and the reinforcement learning-based search strategy is employed to search for the decoder architecture.Experimentally,the searched architectures signif-icantly outperform manually designed architectures on three portrait/face datasets.To further improve the search efficiency and model deployment,a lightweight predefined model is adopted as the encoder,and a multi-objective NAS framework is constructed.The strategies,spanning the two-stage training,feature map pre-computing and memory-resident,are designed to significantly improve the architecture evaluation efficiency.To promote the low-fidelity evaluation effect of lightweight candidate architectures,knowl-edge distillation is imposed to assist the proxy-task training.In this application research,the designed search efficiency improvement and lightweight architectures evaluation strate-gies are worthy of being extended to other application scenarios in the NAS community.