Neural Architecture Search Model Based On Knowledge Distillation

Improvements in computer performance have led to the rise of deep neural network models,however,with the increasing expansion of model sizes,researchers have invested a lot of effort and cost in exploring model architectures that can achieve higher performance with similar parametric numbers.Neural network architecture search is dedicated to automating this discovery process using computers instead of human experts.However,due to the difficulty of evaluating the performance of network structures and the limitations of the weight sharing strategies used in existing efficient models,most of the existing work is based on empirical constraints on the search space to ensure that an approximate solution is obtained efficiently,and it is still an open problem to complete the search for efficient neural network architectures quickly and effectively.In this paper,we design a new search space and search strategy orthogonal to the existing research on neural network architecture search,so that the search space of the neural network architecture search algorithm can be broadened,and by applying knowledge distillation on the basis of the existing search and evaluation strategy,the evaluation of the network structure performance can be more efficient and accurate,and the flexibility of the algorithm in the process of exploring the model structure can be further improved.The main work of this paper includes the following.(1)This paper proposes a tree-type search space that is larger than the linear search space of existing mainstream algorithms and a matching stepwise network search algorithm.Although it is still limited compared to the topological search space searched by the original neural network architecture search algorithm,the structure provided in this paper facilitates the use of a weight-sharing strategy and can maintain a search efficiency similar to that of mainstream methods.(2)In this paper,we propose an alignment method using the knowledge distillation technique to address the assumption of identical functionality between optional structures at the same location for the weight sharing strategy,by which the consistency of the behavior among the optional modules can be improved.The efficiency of the neural network structure search algorithm is improved while the effectiveness of the weight sharing strategy is improved,so that the performance ranking between the subnets inherited from the super-net parameters has a stronger consistency with the performance ranking obtained after the complete training process,thus ensuring that the weight sharing strategy can still have a better effect even in more complex spaces.(3)A method is proposed to adapt the proposed tree-type neural network structure by knowledge distillation technique,so that an acceptable neural network model structure can be obtained quickly by fine-tuning the model structure based on the neural network structure obtained from the existing search when a new condition restriction arises.