Research On Robust Learning Methods Against Noisy Labels In Federated Learning

Federated learning is a privacy-preserving distributed learning paradigm,which can make participating clients to train a global model collaboratively while keep private datasets locally.Federated learning provides a solution for connecting data islands and breaking data barriers,so it has attracted extensive attention from academia and industry.However,obtaining a high-performance global model depends on correctly labeled local datasets,which is a strong assumption in the real world.On the one hand,it is expensive and time-consuming for the clients to obtain perfectly labeled datasets,while the low-cost labeling methods will result in unreliable labels;on the other hand,there may also be malicious clients who will launch poisoning attacks on federated learning by corrupting the labels of local datasets.Such incorrect labels are also called noise labels,and can be divided into non-malicious and malicious noisy labels according to the the reason of their presence.The existence of noisy labels will have a non-negligible impact on federated learning,including reducing the overall model performance or causing the model to misclassify some specific samples.Therefore,how to keep the global model high-performance and robust in the presence of local noisy labels in federated learning is particularly important.In order to improve the robustness of federated learning in the presence of above two class of noisy labels,this thesis conducts the following three research works:(1)To better understand of the impact of the two noisy labels on the behavior of the global model,this thesis analyzes the memorization effect of the global model for the two noisy labels through experiments.Experimental results found that the there is a commonality of the way global model memorize them: the global model itself is robust to noisy labels in the early stage of training,but it will gradually memorize noisy labels as training goes on.And regularization techniques that prevent the model from over-fitting samples can only alleviate the memorization of noisy labels slightly.(2)For non-malicious noisy labels,this thesis proposes a client-side self-guiding method against the noisy labels.This method takes advantage of the robustness of the model to noisy labels in the early stage of training,and proposes to use the output information of historical model trained locally on the client side to guide subsequent training.This method first enforces the model to make high-confidence predictions by sharpening the predictions to prevent the model from memorizing noisy labels.At the same time,the sample-level historical self-ensemble logits are stored on the clients’ device to cancel the storage of the historical model,and the subsequent training of the local model is guided by knowledge distillation of the self-ensemble logits.Comparative experiments on multiple datasets prove that this method significantly improves the robustness of the global model to non-malicious noise labels.(3)For malicious noise labels,this thesis proposes a defense strategy based on federated unlearning.This method saves the gradients updated by the clients during the federated learning on the server,and then extracts a subset of gradient that is highly sensitive to malicious noisy labels based on the memorization effect of the global model as a feature to detect malicious gradients.After the detection,the server first erases the malicious gradients from the final global model,and then calibrates the model deviation caused by erasing the gradient by performing knowledge distillation on an unlabeled auxiliary dataset to unlearn the malicious noisy labels.Experiments on two popular benchmark datasets demonstrate that the proposed method can completely eliminate the global model’s memorization of malicious noisy labels and maintain the model’s performance.