Optimization Theory And Methods For Federated Learning Model Architectures

Due to the increasingly data generation technology,centralized deep learning relying on big data has made significant progress.However,focusing on the real life,with the rapid development of smart devices,distributed mobile terminals have gradually become the mainstream for data generation and storage.But due to privacy protection policies,the data is prohibited from leaving the domain or being accessed by other organizations.This data silo problem poses a considerable challenge to the traditional centralized learning paradigm.In this context,the major service providers have proposed an approach called federated learning.Federated learning emerges as a distributed learning method with privacy protection attributes,facilitating collaborative model training among multiple parties without sharing data.This approach models the distributed terminals by setting up a central server to achieve the common goal of training a global model.Since this process involves no exchange of raw data and encrypts the communication process,federated learning has become a secure and effective solution to address the data silo problem.This graduation thesis thoroughly discusses the challenges of existing federated learning methods and proposes various solutions for issues like communication efficiency,data heterogeneity,privacy and encryption,and model fusion.And from the perspective of applicability,this thesis conducts a modeling analysis of federated learning and introduces a globally generalized universal model.The research in this thesis includes the following parts:(1)To address the contradiction between communication computational efficiency and model accuracy,this thesis proposes a multi-objective federated learning method.In the federated learning scenario,model updates generate a significant amount of information exchange.If the framework directly exchanges models for each training epoch in each communication round,the communication cost would be enormous.This method treats federated learning as a multi-objective problem and employs evolutionary optimization methods for analysis.In this design,neural networks serve as a bridge connecting evolutionary optimization with federated learning,and a spatial search strategy for complex encoding problems is introduced.(2)With the increasing number of users,the problems of data heterogeneity and model heterogeneity caused by device heterogeneity become more prominent.Addressing this issue,this thesis proposes a multi-scale heterogeneous federated learning method.Focusing on the problem of data heterogeneity,the thesis concentrates on different devices holding data of arbitrary sizes and achieves adaptive unification of various feature map sizes by introducing a heterogeneous feature map integration method.To address the issue of model heterogeneity and adapt to devices with various computing performance,this thesis proposes a layer-bylayer model generation and aggregation strategy to customize personalized models for each client.In the aggregation process,shared model parameters are updated layer by layer according to different semantic meanings of network layers.(3)To tackle the challenge of multi-modal data,this thesis proposes a multi-modal vertical federated learning method.Existing federated learning algorithms often overlook the challenges brought by the distribution of multi-modal data.Moreover,previous work relies on a third-party institution for encryption,and encryption has limitations in the exponential and logarithmic operations of the objective function with multiple independent variables.Addressing the challenge of multi-modal data distribution,this thesis introduces a two-step multi-modal encoder model that effectively captures cross-domain semantic features.For the encryption problem,the thesis employs bivariate Taylor series expansion to transform the objective function to meet the limitations of encrypted calculations.Integrating these modules,the thesis proposes a complete training and transmission protocol,eliminating the need for a third-party institution during the encryption process.(4)Combining the above methods,this thesis finally presents a federated contrastive learning method for heterogeneous data sources.The scenario addressed by this model deviates from the constraints of traditional federated learning.This scenario can be described as having any number of participants,where no information is strictly known to each other,but all participants collaborate to complete a global optimization task.This model may be for tasks like video recognition and target classification in the single-modal category,or tasks like crossmodal retrieval and multi-modal understanding in the multi-modal category.In this scenario,the framework first applies the idea of personalized federated learning,guiding each participant to spontaneously generate a feature encoder most suitable for local data.For individual clients,the framework applies the idea from chapter three to classify their input encoders first;for different modal participants,the framework applies the method from chapter four to encode their multi-modal inputs.In the process of model generation and training on the global server,the framework utilizes the idea of contrastive learning to constrain model parameters and generates loss functions based on the global task.This process allows it to fully adapt to various participant data while significantly reducing model complexity.Through a series of experiments,this framework has been proven to outperform other state-of-the-art algorithms in both single-modal and multi-modal tasks.