Compress And Accelerate Deep Convolutional Neural Networks Via Group-based Pruning Methods

Convolutional neural networks(CNNs)have many successful applications in various computer vision tasks.CNNs usually require large computing resources.Many methods of compressing and accelerating CNNs models have been proposed.Network pruning is a effective way to compression CNNs,it includes filter pruning,channel pruning,weight pruning,and kernel pruning.The filter pruning and channel pruning methods remove the whole redundant filters or feature map channels,while,weight pruning and kernel pruning methods prune redundant parameters in a more fine-grained mode: the pruned units are independent weights rather than the whole filters.However,most of the existing methods suffer from degradation of accuracy or the sparse model after pruning,which makes the compression of CNNs difficult.This thesis attempts to compress and accelerate CNNs by using group-based pruning methods.Group-based pruning is a new type of pruning method developed from the traditional network pruning.It has three steps including:(1)clustering the filters in the convolutional layer using grouping criteria,(2)performing the traditional pruning operation within each filter group,and(3)reconstructing the pruned sparse layer into a dense group convolutional layer.Motivated by the deep method of group-based pruning,which has been extensively researched and proven to be effective,group-based pruning is thought to be able to compress the network without losing the accuracy.However,the existing grouping criteria in group-based pruning methods usually use random grouping or traditional clustering algorithms to cluster the filters.These algorithms cannot obtain the relationship among the filters,so that the filters cannot be classified into reasonable groups.In addition,the existing pruning strategies usually use traditional pruning methods and do not propose a more reasonable pruning strategy,which may further reduce the accuracy of the network.To tackle these problems,this thesis will propose three new group-based pruning methods.In this thesis,three novel group-based pruning methods are proposed to optimize the grouping criteria and pruning strategies.The first research is the kernel-principalcomponent-analysis-based group pruning algorithm(KPGP).The KPGP method uses kernel-PCA with a mixed kernel function to reduce the dimensionality of filters;after dimensionality reduction,KPGP utilizes a modified K-Means clustering algorithm to cluster filters into equal-sized groups;subsequently,the parameters are pruned by a normalization-based pruning strategy.The KPGP method demonstrates that the grouping criterion based on kernel-PCA can obtain the relationship among filters and classify filters into reasonable groups.The second research is the group pruning method based on spectral clustering and parametric geometric properties(CSGP).We find that filters are non-Gaussian distributed,and the spectral clustering is more suitable for clustering non-Gaussian data,so CSGP proposes a grouping criterion via modified spectral clustering.This clustering algorithm first constructs the similarity matrix by the k-Nearest Neighbor method,and then constructs the Laplacian matrix by the random walk algorithm,finally clusters the eigenvectors of the Laplacian matrix by the modified K-Means clustering algorithm.In addition,the CSGP method proposes a pruning strategy based on the geometric properties of parameters to replace the traditional pruning strategy based on the normalization.The third research is the dynamic group pruning method based on lottery ticket hypothesis and parameter similarity(TMI-GKP).Inspired by the first two methods and the fact that the parameters in different convolutional layers usually have different data characteristics,we design a dynamic grouping strategy instead of a single grouping method in the TMI-GKP method.TMI-GKP first uses lottery ticket hypothesis to select the appropriate grouping criterion for each layer,and then proposes a pruning strategy based on parameter similarity.The pruning strategy based on parameter similarity can identify the subset of parameters,and this subset contains almost all information and can replace the original parameters.It is experimentally demonstrated that the TMI-GKP method can outperform state-of-the-art pruning methods.In this thesis,the existing pruning methods are optimized by the three different group pruning methods mentioned above,which can compress and accelerate CNNs without accuracy degradation.