Research On Generalization And Algorithms For Federated Learning

If data is the lifeblood of artificial intelligence,then data security serves as its immune system.As a powerful tool capable of breaking ”data silos,” federated learning technology prevents raw data from leaving its local environment.Instead,it achieves multi-party data value release through a joint machine learning modeling approach of ”local storage +distributed learning,” effectively resolving the contradiction between data circulation and data security.Due to its advantages in communication efficiency,large-scale decentralized computing,agile personalized services,and privacy protection,federated learning has garnered increasing attention,and corresponding algorithms have made significant progress.Currently,the development of federated learning algorithms is far ahead of the theoretical aspects; however,there is still a lack of sufficient explanation for why federated learning is effective.The related theoretical research is still in its early stages,especially concerning the generalization theory of federated learning.As a key performance indicator of machine learning models,generalization error measures a model’s ability to adapt to and correctly predict unknown data.Generalization theory is more natural and important in machine learning but seems to have not yet been deeply explored in federated learning.On the one hand,each participant in federated learning has a unique data distribution and samples a limited set of data for collaborative training.Additionally,federated learning consistently faces practical constraints and challenges in communication efficiency and privacy.Therefore,considering such paradigms and real-world challenges,depicting generalization in federated learning is not as direct and obvious as in centralized learning.On the other hand,in practice,the probability of clients participating in federated training is influenced by many factors.The actual participation rate might be low,and some clients may never have the opportunity to engage in the training process.It is precisely these non-participating clients for whom people hope the trained models will be usable.Firstly,addressing the federated learning problem in scenarios with limited communication resources,this dissertation analyzes the generalization properties of the most common technique in communication-efficient federated learning-quantization techniques based on algorithm stability tools.This dissertation proposes upper bounds on algorithmic stability and corresponding tight generalization error bounds for quantized stochastic gradient descent under both convex and non-convex objective functions.Theoretical analysis suggests a trade-off between communication efficiency and generalization error.For convex functions,the generalization upper bound is insensitive to the level of quantization.In contrast,for non-convex functions,the level of quantization significantly impacts generalization,necessitating early stopping of training to ensure the generalization performance of the trained model.Experiments on multiple public datasets validate these theoretical findings.Secondly,to address the challenge of data heterogeneity in federated learning,this dissertation introduces a personalized federated learning framework from a Bayesian probabilistic perspective.By incorporating a global prior distribution,global information is conveyed to local clients.The global prior is designed to capture common intrinsic structures from heterogeneous clients,which can then be transferred to individual local tasks,and simple local updates help clients generate accurate user-specific personalized approximate posteriors.Through theoretical analysis,it demonstrates that the proposed framework provides generalization guarantees for unknown heterogeneous data distributions of clients.Experimental results on various datasets demonstrate the effectiveness and superiority of the proposed method.Thirdly,to address the challenge of a large number of users being unable to participate in federated training,this paper proposes an upper bound on the generalization error for participating users within a hypothetical dual-layer distribution framework,while also characterizing the upper bound on the generalization error for non-participating users.Utilizing the proposed generalization error bounds,we then develop a personalized federated learning algorithm with federated-level regularization and guaranteed generalization performance,effectively avoiding the issue of overfitting to the data of participating users.Finally,extensive evaluations on common datasets demonstrate the effectiveness of the proposed method.