Research And Application Of Efficient Neural Network Structure Search Algorithm Based On Dynamical Isometry Theory

The neural network structure search algorithm(NAS)automatically searches out the best network structure through the algorithm,so as to reduce the artificial experience in the manually designed network.The search cost of existing NAS algorithms is extremely high.In 2022,a NAS search algorithm based on batch normalization layer greatly reduce the cost of such algorithms including search time and calculation consumption.However,there are questionable problems in the design idea of this algorithm,and at the same time,the design principle lacks theoretical proof.Inspired by this algorithm,based on Dynamical Isometry,this paper designs an efficient neural network architecture search algorithm.This paper uses Dynamical Isometry to solve the problem of unfair module evaluation in the NAS algorithm with frozen weights,and proves that when the network implements Dynamical Isometry,the parameters of the batch normalization layer can be used as evaluation indicators to evaluate the feature extraction capabilities of candidate modules in the search space.Finally,the validity and reliability of the proposed algorithm are verified by experiments.The specific analysis will be carried out as follows:1.This paper proposes an efficient NAS algorithm by combining Dynamical Isometry theory with NAS algorithm.First,This paper draws inspiration from the property that Dynamical Isometry can stabilize the forward propagation of the network and implements it in the NAS algorithm with frozen weights to ensure the fairness of algorithm evaluation.In order to use structural parameters to improve efficiency and avoid additional network-independent parameters,this paper redesigns a batch normalization layer-based NAS algorithm based on the realization conditions of Dynamical Isometry.2.This paper provides a theoretical proof of the key ideas and motivations in the proposed algorithm.We first proved that in the NAS algorithm with frozen network weights,Dynamical Isometry guarantee that the search module evaluation process is unbiased.In addition,this paper explains the motivation for using batch normalization layer as a structural parameter in the NAS algorithm,and makes a theoretical analysis of the feasibility of this strategy.3.The effectiveness and feasibility of the proposed NAS algorithm are verified by application.This paper analyzes the changes of the information carried by the feature map in the proposed algorithm,and verifies that it can guarantee the fairness of the search process.Then apply the algorithm to the image classification data set to verify its feasibility.The network structure output on the Image Net classification task can achieve an accuracy rate of 75.76%,and the search efficiency of this algorithm is higher than that of existing research.