A Deep Neural Network Pruning Method Based On Structural Search

Deep neural networks have achieved great success in various vision tasks such as image classification,detection,and action recognition.However,due to the complex structure and high computational complexity of existing deep neural networks,it is difficult to be directly applied to mobile devices and devices with limited computational power.Therefore,neural network pruning,as a tool for model lightweight and computational acceleration,has received attention from academia and industry in recent years and has been widely used in practical deep model deployment applications.The core of deep neural network pruning algorithm is to obtain a compact deep neural network by pruning redundant filter parameters.Most existing deep neural network pruning methods require pre-training of deep neural networks and rely on manual settings.Filtering metrics measure the importance of network structure and pruning.However,such manually designed metrics rely on researchers’ experience and suffer from poor robustness,difficult parameter adjustment,and poor accuracy,and the pre-training process increases the computational complexity of the pruning algorithm.To address the above problems,this paper proposes an automatic pruning method for neural networks based on structure search to obtain compact neural networks directly from redundant neural networks.The specific research contents and contributions of this paper are as follows·In response to the problem that existing pruning methods need to be pre-trained and then pruned leading to a lengthy pruning process and high computational complexity,this paper proposes a fast neural network pruning method based on joint search and training to obtain a compact neural network by searching directly from the beginning.In order to improve the efficiency of pruning,by considering pruning as a search strategy,this paper proposes to prune based on structural search and obtain a compact network structure.Meanwhile,in order to improve the robustness of pruning,this paper adopts a dynamic pruning strategy based on a threshold generation network for automated threshold generation and a knowledge distillation method based on a multi-teacher network.Experiments prove that the method can effectively improve the efficiency of the pruning algorithm without degrading the performance of the pruning algorithm.·To address the problem that the pruning efficiency of existing pruning methods decreases substantially at high pruning rates,this paper proposes a filter pruning method based on network structure expansion search.Compared with the traditional pruning method of finding filter combinations in a fixed network space,this paper advocates the strategy of "expansion before pruning" to improve the efficiency of structural search.Specifically,this paper first constructs an extended search space and introduces a global group sparsity coefficient based on a Gaussian scale mixture model to measure the importance of filters,and finally prunes the network iteratively by a novel deterministic annealing strategy.Through research and extensive validation experiments,this paper finds that by combining the expanded search space with global sparsity,the pruning method based on structural search can search for a more reasonable compact neural network structure.The proposed method outperforms existing pruning methods in terms of parameter size reduction of compact networks and performance under high pruning rates,and is a pruning technique particularly suitable for practical deployment applications.·Iterative pruning is a class of pruning techniques that gradually prunes the network parameters,and it has gradually become a mainstream pruning strategy due to the controllable pruning loss and high accuracy.However,the existing related pruning methods ignore the interaction between adjacent rounds in the iterative pruming process,thus accumlating errors and causing additional performance degradation in the pruning process.To address this problem in iterative pruning,this paper proposes an iterative pruning method with pruning and reparameterization by Bayesian estimation.In each pruning round,this paper first estimates a Bayesian model based on the results of the previous pruning rounds and predicts the pruning probability distribution of different channels based on this model,and the predicted probabilities are used for pruning.In the specific pruning process,the pruned network is also reparameterized based on the probability distribution in this paper in order to reduce the accuracy loss from pruning.Experimental results on several popular datasets show that the proposed method can accurately predict the pruning results during the iterative pruning process,effectively improving the efficiency and accuracy of pruning.