A Study On Search Space Optimization-based Differentiable TextNAS

Features extracted from discrete text data for text classification tasks will vary greatly depending on the domain and background,making text classification networks more complex.Such complexity increases the difficulty and cost to find the optimal text classification network model manually.Neural Architecture Search(NAS)automates the network model design process,which reduces the labor cost of text classification model design.The current NAS algorithm used for text classification tasks has two problems.First,modeled as Markov Decision Process(MDP),NAS is a task with delayed rewards,which slower the optimization.Second,search space settings of NAS relies on human experience.Unfiltered search space settings will reduce the accuracy of the network model designed by the NAS and the efficiency of NAS.To solve the first problem,we replace the MDP modeling in NAS with a graph convolutional network and reparameterization trick to generate the network structure,so that the derivative of the graph convolutional network weights to the validation losses is computable and accelerates the optimization.In the text classification experiment,compared with State-Of-The-Art(SOTA)NAS algorithm based on MDP modeling,the proposed algorithm reduces the time consumption of searching for the optimal model by 39% while the accuracy is improved by about1% to 6%.To solve the second problem,we model the search space selection problem as a subset selection problem and solve it with an evolutionary algorithm,which replaces the complete search space with a smaller subspace to reduce computing resources required by NAS.In the text classification experiment,compared with the original NAS method,the improved algorithm reduces GPU memory usage by 25% and the search time by 35% under the premise that there is no significant statistical difference in the predicted value.To apply the above algorithms to text classification tasks,we designed a NAS framework and NAS system,providing automatic search space selection and NAS services for text classification tasks.We choose 3 open source text classification models for testing of the NAS system.For the same input,there is no significant statistical difference between the output of the text classification models designed by the NAS system and the manual design models,indicating that the automatically designed model by the NAS achieves the same text classification performance as the manually designed ones.By automating the design of the network models,our NAS system allows engineers who are not equipped with knowledge about deep learning to design text classification network models and reduces the difficulty and labor cost of the design work.