Lightweight Networks Research Based On Pruning

In recent years,deep convolutional neural networks have achieved impressive results in various artificial intelligence applications.On the one hand,excellent algorithm performance is based on the huge number of parameters and computational overhead.On the other hand,there is a growing demand for training and deployment of models on edge devices(e.g.,mobile terminals,small unmanned devices).This leads to the fact that the currently designed networks and their training strategies are not well adapted to the characteristics of edge devices.For example,in a scenario where multiple devices are trained together,how to train models when communication bandwidth is limited and how to deploy edge models when inference resources are limited are both challenging issues.To tackle these challenges,lightweight networks have received a lot of attention from academia and industry.Pruning techniques that remove noncritical or redundant parameters from the models are one of the important strategies.This dissertation thoroughly investigates and studies the application of this technique in different scenarios.In the lightweight deployment of the network,this dissertation improves the previous method of parameter pruning based on the empirical preset fixed pruning rate according to the characteristics of the network and proposes an automated pruning rate setting strategy to efficiently remove the redundant parameters in the network and achieve a balance between network performance and computational effort.In the collaborative training of the network,this dissertation extends the pruning object to the transmitted model weights/gradients by extending the pruning technique,which significantly reduces the number of parameters exchanged between the clients and the server and reduces communication overhead.The main research and innovation points of this dissertation are:1.This dissertation proposes a general dynamic connection pruning algorithm to evaluate the importance of parameters and the pruning rate of convolutional layers/blocks in order to efficiently and reasonably eliminate redundant connections in densely connectednetworks,taking the classification and super-resolution tasks as examples.First,the algorithm proposes a sampling-evaluation process to evaluate the contribution of connections to the network by iteratively picking the sub-network and evaluating its performance.Connections with low contributions are pruned in a subsequent process.Second,to determine the pruning rate of the convolutional layers/blocks,the algorithm is based on a Markov process that prunes the low-contributing connections one by one until the process terminates.The state transfer probability of the Markov process is embedded in the loss function,which guides the training process together with the model performance loss function.The proposed algorithm not only evaluates the connection importance but also automates the learning of the pruning rate of each convolutional layer/block.Experimental results demonstrate that the proposed algorithm achieves a good trade-off between network performance and computational effort,reducing the computational effort of Dense Net-40 by half with almost no loss in accuracy?2.This dissertation proposes a dynamic update algorithm for spatial-temporal dimensionality for federated learning to address the problems of data heterogeneity and communication bandwidth limitations in multi-device collaborative training.The algorithm exchanges pruned lightweight model weights/gradients between the clients and the server,and the pruning rate(namely the update rate in federated learning)of the model is dynamically updated.First,the algorithm dynamically assigns update rates to the model at different communication rounds based on historical training information.Second,the global information required varies from client to client even in the same round.The algorithm iteratively groups the clients according to the model weights,and the clients in the same group share a common update rate.In this dissertation,experiments are conducted to compare the state-of-the-art method with IID and non-IID settings for three benchmark datasets.The algorithm dynamically assigns update rates to each client at different training stages.The experimental results demonstrate that the method can effectively improve the accuracy of the federation learning system while reducing communication overhead.For example,on the FEMNIST dataset,the method improves the accuracy ofthe backbone network by 10.93% and reduces the communication overhead by 95% with the traditional federal learning method.3.This dissertation proposes a delayed dynamic update algorithm for high latency federal learning scenarios to simultaneously solve the network delay and communication bandwidth limitation problems.To solve the bandwidth limitation problem,the algorithm requires the clients to upload the pruned model gradient parameters and perform partial gradient averaging at the server,and the number of parameters of the uploaded partial gradients(controlled by the pruning rate)changes dynamically with the training process.In addition,in order to reduce the communication waiting time,the algorithm executes the communication and local computation in parallel,which mitigates the impact of latency on the convergence speed.Also,the algorithm introduces a gradient correction term to compensate for the errors arising from parallel execution.In this dissertation,our method is evaluated under two benchmark datasets with IID and non-IID settings.The proposed federal learning training method performs communication and local computation in parallel on the basis of dynamic update rate,which greatly improves the model convergence speed.For example,in the CIFAR-10 setting,which has the highest degree of heterogeneity,our method requires only 5 units of runtime to achieve the target accuracy,while other methods require from 38 to 97.