Efficient Federated Learning Research And System Development For Data Heterogeneous Scenarios

Federated Learning(FL)enables multiple parties to train a global model collaboratively without sharing local data.However,a key challenge of FL is data distribution heterogeneity across participants,which causes model drift in local training and significantly reduces the performance.In addition,when federated learning faces large-scale scenarios with massive devices,the expensive communication overhead can be a significant training bottleneck,especially for devices with limited communication bandwidth.Therefore,how to improve the convergence performance and reduce the communication overhead of federated learning algorithms in Non-IID scenarios is an essential topic in federated learning research.To address the above issues,the main contributions of this thesis are as follows:1.We analyze the inconsistency differences between different local model layers of local models and propose the Layer-wise Distance Regularization(LWDR)method to improve the federated learning local training process.It uses different distance regularization constraint weights according to the depth of the model layers,which accommodate the inconsistency differences between different layers of the local model.This method can improve the performance of local training.2.For the stability of model aggregation in large-scale FL scenarios,this paper further applies the idea of layer-wise re-weighting to the process of global model aggregation.We propose the Layer-wise Momentum Aggregation(LWMA)method to improve the model aggregation process in large-scale scenarios.This method introduces momentum in the model aggregation process,where the momentum is weighted according to the depth of the model layers.This method can improve aggregation stability while reducing the impact of momentum on the model convergence speed.3.To support multi-device collaborative training,we design and implement a federated learning system for Io T scenarios based on the Io T communication technology MQTT and the web service framework Flask.The system ensures scalability,fault tolerance,and communication stability through mechanisms such as dynamic scaling,heartbeat detection,interrogation response transmission,and timeout detection.In addition,to reduce the communication overhead of devices,this system designs and deploys a gradient compression method to improve communication efficiency and reduce the communication cost of the system.To verify the effectiveness and usability of the algorithm and system,we conducted a series of federation learning experiments based on CIFAR-10 and CIFAR-100 datasets.The results show that our method can effectively improve the convergence performance of the model in the Non-IID scenario,and the federated learning system in this paper has high effectiveness and usability.