Research On Communication-efficient Federated Learning Algorithm On Heterogeneous Dataset

With the popularity of mobile devices,a large amount of clients’ data has been generated at the edge,which brings countless possibilities for various machine learning applications.Due to the introduction of relevant data protection regulations,it is impractical to collect all clients’ data in the cloud for training,so federated learning is proposed.Clients participating in federated learning completely keep the data locally and complete the training only through multiple local updates and model parameter exchange,so as to achieve the purpose of privacy protection.Clients participating in federated learning often have obvious behavior differences,so their dataset distribution is also inconsistent.Heterogeneous datasets lead to divergence of clients’ model during local update,which requires more communication rounds to achieve convergence in many times.Frequent model parameter exchange will bring huge communication overhead,and the client nodes participating in training often have limited communication resources,so the communication efficiency of federated learning has always been a hot research issue.In federated learning,the strategies to improve the communication efficiency mainly include reducing the communication frequency and model parameter compression.One method to reduce the communication frequency is the local stochastic gradient descent algorithm,which uses the method of performing multiple local updates for each client to avoid frequent communication.The common method in parameter compression is parameter quantization,which uses the mechanism of function mapping to reduce the communication bits required for model transmission.In the existing literature,the effectiveness of these two strategies has been confirmed,but there are still some deficiencies:1)In local stochastic gradient descent algorithm,clients need to compute the stochastic gradient for local update.It can not be applied to some scenarios where the gradient information is not available,such as black box attack and hyperparameter optimization.2)The existing methods often set the hyperparameters through empirical or heuristic rules.Using less transmission bits and more local update times can significantly reduce the communication overhead,but will affect the final accuracy.Using more transmission bits and less local update times can improve the final convergence accuracy,but there is obvious communication redundancy,which will lead to performance degradation to varying degrees.This work studies how to improve the computation and communication steps in federated learning with heterogeneous datasets,and improve the final convergence accuracy on the premise of ensuring the communication efficiency of the algorithm.The main research work is as follows:1.A communication efficient decentralized zeroth order algorithm under heterogeneous datasets is proposed,and its convergence performance is analyzed.We introduce the more practical zeroth order algorithm into the decentralized federated learning scenario.It does not need to use first order information,but only needs to measure the loss function values locally.We use the strategy of periodic parameter exchange in local stochastic gradient descent to design the algorithm,and analyze the convergence performance of the algorithm on heterogeneous datasets.At the same time,we clearly characterize the influence of local update times and zeroth order estimation error on the convergence rate,which provides a theoretical basis for federated learning training under heterogeneous datasets.Finally,our theoretical results are verified by simulation.When reaching the same convergence error,the communication rounds required by the decentralized local zeroth order algorithm are significantly reduced compared with the existing distributed zeroth order algorithm.2.A fast federated learning algorithm based on adaptive gradient transmission is proposed.We study the influence of two key hyperparameters,parameter quantization level and local update times,on the communication bits required for training,and use Markov decision process to formally model the problem,design an adaptive gradient transmission strategy.It uses the training information collected from the past communication rounds to determine the frequency and parameter quantization level,automatically balance the communication overhead and convergence accuracy,and alleviate the disadvantages caused by artificial hyperparameter adjustment.The simulation results on real datasets show that the joint dynamic selection of these two hyper-parameters can achieve lower convergence error under the same communication bit overhead.