Sparse Optimization And Searching Of Deep Neural Network Structure

The rapid development of deep learning has made the methods of neural networks gradually replace traditional pattern recognition methods,of which explosive applications and leading progress have been made in many fields.Despite the leading performance,the advantages of deep learning come from the big data,complex models,and sufficient computing power.It is still vague how to select the right network structure to solve a task,which means the exact model structure and scale required to complete a task can be not accurately obtained.Most tasks always rely on universal complex models.The training of complex models will consume great computing resources,and at the same time,the deployment of the complex model will also be subject to many restrictions.Therefore,in the design of a model,the way to compress the over-parameterized model into a smaller one,namely,the sparse optimization of the network structure has gradually become the focus of research.Introducing the sparseness can reduce the calculation of the model and provide the possibility of model deployment to edge devices,expanding the application scenario of deep models.In addition,the sparsity of the network can provide a new foundation for theoretical research of deep learning.Another promising approach to network structure optimization that breaks through this limitation is neural network structure search,which is different from network sparsification and focuses more on performance enhancement by matching the model structure changes with the task.This thesis addresses the challenges and the shortcomings of existing works on network sparse optimization and network structure search,and provides an in-depth study of network structure sparse optimization based on search and multi-objective optimization,the training of sparse networks,the application of network sparsification in multi-party learning,and the application of network structure search in remote sensing image classification tasks,which can be summarized as follows:(1)A method of adaptive network sparse optimization based on a layer-wise search strategy is proposed by combining the two through the study of network sparse optimization and network structure search.In response to the inability of existing sparse optimization methods to adaptively adopt independent sparse criteria for different network layers,the idea of structural search is adopted to perform hierarchical sparse optimization for complex neural networks.In the sparse optimization process,a set of feasible candidate sparse strategies is selected layer by layer,and a corresponding weight is assigned to the strategy,which is gradually optimized with the network training,and then the optimal sparse strategy for the layer is obtained.A layer-adaptive sparse strategy is executed for each layer,and finally a sparse network with layer-adaptive properties is obtained.(2)Through decomposing the neural network sparse optimization problem into two conflicting objectives of performance and scale of the network,it is transformed into a multiobjective optimization problem,and a multi-objective evolutionary algorithm based on the decomposition is used to continuously optimize these two conflicting objectives.Meanwhile,an input adaptive mechanism describing the mapping relationship between input data and network structure is established to construct an agent model of the network,accelerate the performance evaluation of the sparse network,and effectively reduce the time complexity of the evaluation of the sparse network.(3)A self-decoupled sparse optimization strategy is proposed by rethinking the existing sparse optimization approach for neural networks.Generally speaking,the complete network always obtains higher performance than the sparse backbone network due to the complexity of the structure leading to stronger feature extraction.In the strategy adopted in this thesis,instead of focusing excessively on the sparse strategy adopted for the neural network,we target the sparse training process of the network more.The neural network is divided into two parts,the backbone network,and the redundant units.A neural network is divided into two parts,the backbone network,and the redundant units.The two are separately optimized,in order to optimize the backbone network with the information of redundant units during the training process and to continuously decay the redundant units to finally obtain the sparse backbone network.This strategy can be implemented in a randomly selected backbone network without considering other special sparse strategies,or the sparse network obtained by other sparse strategies can be re-optimized to obtain better performance.(4)In a specific application,the idea of network sparsification is applied to multi-party learning,and a sparse optimization strategy for multi-party learning networks based on contrastive distillation is proposed.In the multi-party learning task,due to the heterogeneity of local data,the local model can be considered as an unbalanced sparse sub-network in the global model.By executing sparse optimization strategies among individual local models,it is possible to train sparse subnetworks for different user data heterogeneity,enhance the convergence of the aggregation network,and improve communication efficiency in multi-party learning.Through contrastive distillation,we are able to enhance the connection between the local model and the global model,so that the local model can effectively extract the local data while maintaining the offset between the local model and the global model due to data heterogeneity.Finally,an efficient and sparse multi-party learning model is obtained.(5)For the specific task of remote sensing image classification,a two-stage multi-objective neural network structure search algorithm is proposed.This algorithm divides the traditional evolutionary multi-objective network structure search algorithm into two different stages,which can significantly reduce the space of network structure search,and improve the efficiency of network search.Through the two-stage method,we can obtain a sparse structure by striking a balance between the performance and structure of the model.It has achieved good results in the remote sensing image classification task.