Research On Neural Architecture Search Based On Differentiable Strategies

Neural architecture search is gaining popularity in the field of machine learning as an automated design method.Up to now,various neural network architecture search methods have made great progress,and the neural architecture search based on differentiable strategies stands out among many search algorithms by virtue of its low search cost and easy implementation.Although this search strategy has many advantages,more in-depth research is needed regarding the gap between the search stage and the evaluation stage and the control of the number of search iterations in the search stage.Moreover,both the search network and the evaluation network are formed by repeatedly stacking Cells,which is also a bottleneck that limits the performance of the network.The above disadvantages lead to certain constraints on the searched architecture,making the search results not optimal in the evaluation stage.In order to solve these problems,this paper conducts the following research on the search and optimization of the architecture:Aiming at the problem that there is a gap between the search and the evaluation stage in the progressively differentiable architecture search algorithm,which leads to unstable search results,and the problem of excessive search time overhead due to invalid search.This paper proposes to gradually mitigate the initial-channel gap to bridge the channel gap between search and evaluation while relieving the excessive growth of search cost,The gap between the search and evaluation is a common problem in current differentiable architecture searches.P-DARTS narrows the depth gap by gradually increasing the depth of the search network to approach the evaluation depth,However,since there is still a large gap in the number of initial-channel for the network,it is difficult for the searched architecture to be optimal.For this reason,this paper proposes a method of gradually mitigating the initial-channel gap,and adopts a search space separation strategy to alleviate the amount of calculation caused by increasing the number of channels.In addition,in order to further improve the search efficiency,this paper explores the relationship between the number of search iterations and the stability of the architecture,and proposes a strategy to stop the search in time.Experiments show that on CIFAR-10 and CIFAR-100,the test errors are only 2.44%and 16.34%,respectively,and in the same environment,the search time is only 0.7 of PDARTS.Aiming at the problem that the architecture needs to repeatedly stack Cells in the differentiable architecture search,which leads to the limitation of network performance,this paper proposes an optimization strategy based on the differentiable architecture search algorithm.In the general search algorithm,no matter in the search stage or in the evaluation stage,two kinds of Cells are often designed,such as Normal Cell and Reduction Cell.These Cells are then stacked repeatedly to form the super-net during the search stage or the evaluation network.This repeated stacking method restricts the performance of the architecture,resulting in the Cells obtained in the search stage,and the evaluation network formed by stacking repeatedly may not be optimal.In order to break through this constraint,this paper regards the evaluation network formed by stacking repeatedly of the searched Cell as the network to be optimized.Experiments have shown that by combining the DARTS first-order and second-order algorithms with the optimization algorithm proposed in this paper,our test errors on CIFAR-10 dropped from 3.03%and 3.06%to 2.54%and 2.58%,respectively,and the test errors on CIFAR-100 dropped from 17.76%and 17.54 to 15.68%and 15.73%,respectively.