Efficient Edge Federated Learning By Using Dynamic Sparse Training

Nowadays,billions of edge devices are connected to the Internet and generate a massive amount of data,which can be used to train more accurate models for complex tasks.How to fully mine and utilize this edge data has become a hot topic.Traditional methods need to collect all data together and then perform centralized training,but this process has unacceptable risk of privacy leakage.As one of the most promising solutions to fully exploit the data on edge devices,Federated Learning(FL)can combine data from multiple parties and train a powerful global model while protecting clients’ data privacy.However,the complexity of deep learning models brings unbearable pressure to resourceconstrained mobile terminals and edge devices,which greatly limits the practical deployment of FL in mobile edge computing systems.In addition,FL faces the challenge of data heterogeneity in practical applications,which makes the training of the global model very difficult or even impossible to converge.In order to solve the challenge brought by data heterogeneity and fairness considerations,a new paradigm of personalized Federated Learning was appeared.In order to make it possible to use Lederated Learning to mine edge data on resourceconstrained edge devices,this thesis aims at resource-constrained mobile terminals and edge devices,combined with the intelligent computing and personalized service requirements of mobile edge computing scenarios,researches on mobile edge devices.This thesis tries to enhance the expressibility of sparse training in Federated Learning,and improves the efficiency and applicability of sparse training in distributed systems.Unlike existing works which use expensive dense models,this thesis proposes two different paradigms to improve the effect of federated learning sparse training:(1)By proposing a new sparse training concept of parallel over-parameterization,a powerful global sparse network is trained to ensure that the global model can better generalize Convert it to any device,so that it can be used”out-of-the-box";(2)Search and train a unique personalized sparse sub-network for each client,and give each client a higher return than the global network,to achieve"better than better".Specific contributions of this thesis includes:1.Sparse-to-Sparse Traing by Parallel Over-Parameterization.This thesis explores the feasibility of training a sparse model from scratch in Federated Learning(sparse-to-sparse training).In order to narrow the gap of fitting ability between the sparse model and the original dense model,this paper proposes a new concept-Parallel Over-Parameterization(POP),in order to keep the performance of the model with the goal of reducing computational and communication overhead at the same time.2.POP-FL:Towards Efficient Federated Learning on Edge Using Parallel OverParameterization.Designed for sparse training of the global model.To address the challenge of performance loss of sparse training(due to poor expressibility),this thesis dynamically optimize the sparse network structure during training the network parameters instead of fixed beforehand.This thesis embed POP in FL and propose a novel framework called POP-FL.To make better use of the distributed characteristics of FL systems,POP-FL divides massive clients into groups,and performs POP in the dimensionality of space manifold,i.e.,reliably exploring parameters of the original model over the collaboration between groups in parallel.The underlying rationale behind POP-FL is that dynamically updating the global sparse network’s structure from an explored POP parameter space can enhance the expressibility of sparse training in FL.This thesis conducts extensive evaluation and experimental results show that POP-FL outperforms existing pruning-only and DST methods on various representative networks.3.Adaptive Sparse Neural Network Exploration Enables Efficient Personalized Federated Learning.Designed for sparse training of personalized local models.For the first time,this thesis studies the use of adaptive sparse network structure search to realize personalized Federated Learning,and achieves the goal of significantly improving the performance of local models through personalized methods under extreme data heterogeneity.This thesis proposes ASNO-PFL,which allows each client to dynamically optimize its own sparse network according to local data,so as to find a personalized sparse network that is more suitable for its own data distribution.This thesis validates the effectiveness of ASNO-PFL on three datasets of different scales,and the experimental results show that ASNO-PFL outperforms current pruning-based personalization methods.