| Federated Learning has emerged as a promising distributed machine learning paradigm,the main reason of which is its huge privacy-preserving potentials.However,one of the fundamental challenges facing Horizontal Federated Learning is that the data stored on different clients’ devices are usually distributed in a non-independent and non-identical way(a phenomenon also referred to as data heterogeneity).In this thesis,four different scenarios of heterogeneous data distributions are simulated by data partitioning,each of which corresponds to one type of data heterogeneity.Through experiments run on these partitioned data,it is shown that the classic Federated Averaging algorithm fails to address the issue of client drift effectively and thus suffers both a loss in its convergence speed and in its global test accuracy.It exacerbates the problem even further when increasing the number of local epochs.To tackle data heterogeneity,a feature generation method based on Generative Adversarial Networks(GANs)is proposed for the aim of classifier correction.During each global epoch,users first train their GANs independently to learn the distributions of their real intermediate features.Their generators are later sent to and used by the server to generate fake intermediate features and to correct classifer biases.Experiment results show that this strategy can boost the convergence of the single-centre global model,reducing the number of communication rounds needed to reach its highest test accuracy.Meanwhile,to deal with the particular problem of concept shift,a personalised model aggregation approach is proposed on the basis of classifier clustering.Before model aggregation,the server divides users into several groups through clustering of their classifiers.Thereafter user models are aggregated according to the clustering results,forming multi-centre cluster models which help users achieve better personalization.The two strategies mentioned above are then integrated and tested.Through visualisation of empirical results,this integration is shown to be effective in realizing both better generalization of the single-centre global model,and better personalization of the multi-centre cluster models.Finally,to alleviate client drift caused by increment in the number of local epochs,a regularization method utilizing response-based knowledge distillation is proposed.It is demonstrated experimentally that this method helps the global model converge in a way that is smoother,and in some cases,better and faster. |
