Model Update Optimization For Heterogeneous Clients In Federated Learning

Federated learning is a distributed machine learning framework that uses a server to coordinate multiple resource-constrained clients to train a shared model.During the training process,clients do not need to transmit the user's local data and only needs to transmit model update information to jointly train the prediction model.This decentralized model training method has the advantages of protecting user data privacy and low transmission cost.At present,federated learning has received attention in both research and application fields.Limited by the server's bandwidth,in order to ensure the training efficient,the server cannot wait for the all gradient updates to be uploaded,and can only collect partial gradient updates.Therefore,rational use of communication resources and improvement of transmission efficiency are very important to improve the performance of federated learning.Affected by clients heterogeneity,there are differences between client data,so the effect of training on different clients may be quite different,and these different training results also have different effects on the training of the overall model.Even if the number of gradient updates collected by the server is the same,the effect of updating the global model may be different because the update gradients come from different clients.In order to meet the challenge of limited communication resources in the federated learning environment,this paper proposes a client updates selection strategy based on client training value to improve the federated learning algorithm.The client training value evaluation is based on the current model and client training samples.In each round of training,the server selects clients with greater training value to upload gradient updates.This solution will prevent some clients with low training value from participating in the current model training,and reduce the communication frequency between these clients and the server,so that the server's communication resources can be effectively allocated to clients with high training value.In the case of collecting the same number of gradient updates,this method can speed up model training.Experiments show that this scheme can reduce the number of communication rounds by at least 30% on the experimental dataset to achieve the same convergence.In federated learning,the gradient update is affected by the heterogeneity of the client data and diverges too much,causing the aggregated global model to be far from the global optimum.This is the most direct factor in the performance degradation of Non-IID data.This paper introduces the update divergence to measure the divergence of the client gradients updates,and proposes a new framework,Fed Norm.On the one hand,this algorithm introduces a new objective function for client training.The objective function adds a regular term to suppress the divergence of client gradient updates.On the other hand,it provides clear and consistent optimization goals for client training by constructing a training dataset on the server side.In the experiment,the scheme showed a more stable and fast convergence process compared to FedAvg and FedProx.