Research On Performance Optimization Of Federated Learning For Data Heterogeneity

Federated learning(FL)enables all the involved Internet of Things(Io T)devices to collaboratively train a global model remaining the original data distributed,which can achieve an efficient distributed machine learning paradigm with low communication overhead and data privacy protection.However,the heterogeneity of the data collected by the Io T devices will lead to the “weight divergence” in the FL model training process,which will slow down the model convergence rate and reduce the model accuracy,leading to serious security risks.Although there exist various FL optimization algorithms that try to tackle this problem,they will introduce a lot of communication overhead or expose the privacy of their original data,which makes them unsuitable for Io T applications with strict data security requirements and resource constraints.Therefore,how to improve the FL performance for data heterogeneity without introducing intolerable communication overhead and exposing data privacy is becoming a major research issue for FL in Io T scenarios with data heterogeneity.To address the above challenges,this thesis designs three different optimization methods around the three processes of FL model training(i.e.,the device selection process,the local training process,and the cloud aggregation process)to improve the FL model performance with data heterogeneity.To be specific,the main research contents and contributions of this thesis are as follows:(1).Based on the similarity of device data distribution,a novel FL device selection approach that relies on device grouping is proposed to optimize the device selection process.This thesis uses the pre-trained model in the library to extract the data features of each device,thereby obtaining the features of all participating training devices.After that,all the devices can be clustered using the hashed device feature maps without exposing data privacy.In this way,this thesis can restrict the device selection in the FL model training process,so as to solve the problem of “weight divergence” and model convergence oscillation caused by data heterogeneity.(2).Based on knowledge distillation,a novel FL local training approach that dynamically adjusts the soft targets and the hard labels is proposed to optimize the local training process.In this thesis,all the selected devices use the updated local models to calculate the label-wise sample logits after each round of local training and upload both the model gradients and the label-wise sample logits to the cloud server for aggregation.All the selected devices receive both the global model and the soft targets from the cloud server and conduct the local training with the designed loss function and dynamic adjustment strategy.In this way,the problem of low knowledge mapping efficiency during the local training process can be solved, thereby improving the model accuracy.(3).Based on device concatenation,a novel FL cloud aggregation approach including device group and device counting methods is designed to optimize the aggregation process.This thesis divides the FL model training process into multiple model training cycles and delays the frequency of global model aggregation in the cloud server.To encourage more devices to participate in the model training and avoid training the same model copy on the same device,this thesis groups all the devices and counts their participation.By periodically aggregating all the model copies that traverse multiple devices,the proposed approach can effectively solve the problem of small training set size and skewed training data,thereby improving the FL performance.This thesis uses multiple well-known benchmarks,data heterogeneous partitioning settings,and classical models to verify the performance of the proposed approaches.The experimental results demonstrate that the proposed approaches can improve three different FL model training processes(i.e.,the device selection process,the local training process,and the cloud aggregation process)without introducing huge communication overhead and exposing data privacy.The proposed approaches can effectively improve the model accuracy and speed up the model training rate.