Research On Neural Architecture Search Based On Knowledge Distillation

Neural architecture search aims to automatically design network architectures by machines,which is expected to bring about a new revolution in machine learning.Despite high expectations,the effectiveness and efficiency of existing neural architecture search solutions are unclear,and the inefficiency of solutions is mainly attributed to the excessive computational cost required to accurately search and evaluate network architectures.Since knowledge distillation technology has received more and more attention,it can compress large models and transfer knowledge to small models.Therefore,knowledge distillation can be introduced into neural architecture search to achieve lowcost research.However,how to transfer the knowledge of the teacher model to the student model is the key problem in knowledge distillation.It is also hoped that the accuracy of the student model exceeds that of the teacher,and the current network architecture as a teacher model has only a maximum accuracy of more than 70%.Therefore,this paper aims at the knowledge distillation search model.To solve the problem that the teacher model is not completely accurate,a dual-loss block-based neural architecture search module(DLB)is proposed.The DLB module is guided by the teacher model and combined with ground truth to supervise the neural architecture search with a double loss function.In addition,the large search space of the neural architecture search is divided into multiple modules,which not only effectively reduces the error caused by the shared parameters,but also ensures the candidate architectures in the module are fully trained,and this block search method can also evaluate all candidate architectures in a module,solving the problem of inaccurate evaluation.In addition,according to the cost problem in neural architecture search,the research direction focuses on the design of the search space,and then this paper also proposes a deep cell search space(DCSS)in the key block mode,which regards the search space as a resource reallocation problem.By identifying the location of key blocks and prioritizing resource allocation to key blocks,the DCSS method is implemented,which effectively realizes neural architecture search and maximizes resource utilization.Finally,this paper also combines the DLB module and the DCSS method to search for the Dlb-Dcss NAS network,which achieves the highest accuracy of 77.4% on Image Net under the same computational constraints as other models,which is higher than the efficient network Efficient Net-B0.1.1%,which is0.4% higher than DNA.Experiments show that the Dlb-Dcss NAS network proposed in this paper significantly improves the effectiveness of neural architecture search and achieves efficient and high-precision neural architecture search.