Deep Model Compression With Neural Architecture Search

In recent years,deep learning methods have made great success in many research areas.However,the deep learning models have a large number of parameters and computations which limit their real-world applications.So,the model compression is of great significance to the applications of deep learning.Network pruning and compact network design are two commonly used methods for model compression,which can effectively reduce the model complexity.However,there are still some problems in these methods,such as the three-stage process of the traditional pruning method requires a large number of computations and is very time-consuming,and the fixed pruning rates greatly limit the flexibility of the methods.While for compact network design methods,designing a model manually needs a lot of trialand-error costs,which may be sub-optimal.However,the models designed with the neural architecture search(NAS)methods are always for the high-level visual tasks,few of them are designed for low-level visual tasks like image super-resolution.Facing the above problems,we proposed a network pruning method based on the NAS and a lightweight super-resolution method based on NAS.Our main contributions are summarized as follows:1.To solve the problem of traditional three-stage pruning process,we propose a joint searchand-training network pruning method which significantly reduces the computing cost of pretraining the large models in the traditional three-stage process.We directly search for the architecture of the sub-network from the scratch and considering the filter pruning as a part of the search process.Aiming at the problem of fixed pruning rate,we propose an adaptive threshold strategy based on reinforcement learning(RL),which can flexibly adjust the number of the filters during pruning.We propose the Thres Net as the agent of RL,which automatically decides the pruning rate according to the distribution of the architecture parameters.Experimental results show that the proposed pruning method achieves great performance on multiple models and multiple datasets,and remarkably reduces the consuming time of the pruning methods.2.To solve the problem of the trial-and-error costs in handcrafted methods,we introduce the NAS method into the low-level visual task as image super-resolution,and design the search space on both the cell-level and the network-level.Specifically,the cell-level search space is designed based on an information distillation mechanism,focusing on the combinations of lightweight operations.The network-level search space is designed to consider the feature connections among the cells,and aim to find which information flow benefits the cell most to boost the performance.Different with other NAS methods which based on RL or evolutionary algorithm(EA),our method is totally differentiable and the computation cost of our method is significantly lower.We design a loss function that considers distortion,highfrequency reconstruction,and lightweight regularization that push the searching direction to explore a better lightweight SR model.Extensive experiments show that our method surpasses all the handcrafted and NAS-based lightweight image super-resolution models,and achieves state-of-the-art performance on the benchmark datasets in terms of PSNR,SSIM,and model complexity.