Research On Efficient And Secure Federated Learning Based On Gradient Compression

Faced with data islands and data privacy issues that impede the development of artificial intelligence,federated learning has evolved as a distributed machine learning system that protects privacy.Its advantage is that collaborative modeling can be accomplished without extensive data transfer.In contrast to typical centralized machine learning,federated learning necessitates frequent communication between the client and the server and a significant number of model parameter exchanges,resulting in a substantial communication overhead.With the rapid expansion of the data(model)scale and the broad deployment of federated learning on devices with limited communication and computing capacity,the issue of excessive communication overhead in the training process of federated learning has become increasingly important.Gradient compression and adaptive gradient aggregation are prevalent techniques for enhancing the communication efficiency of federated learning.However,both techniques only examine the dimensions of compressed uploaded gradients and impose an additional computational burden on the client.The optimization is insufficient.Alternatively,gradient leakage is an issue in federated learning.With the uploaded gradient information,malicious participants or third parties can conduct privacy attacks such as membership inference attacks,model inversion attacks,etc.,and then infer or reconstruct sensitive information,ultimately leading to the disclosure of the original data set.In federated learning,differential privacy technology is a mainstream technology.The implementation of differential privacy technology protects gradients without compromising the communication efficiency of federated learning.Nevertheless,the server combines the high-dimensional gradients with noise from the client,which multiplies the privacy loss and reduces the model’s precision.How to secure gradient parameters and assure the efficacy of federated learning communication is a pressing issue in the implementation of federated learning.This work examines the implementation of gradient compression,adaptive gradient aggregation,and differential privacy technologies in federated learning.Theoretical analysis and simulated distributed trials demonstrate that the communication efficiency and data privacy of the strategy presented in this study is exceptional.The application of learning technologies in several fields improves communication security and privacy protection.The following are the specific contributions of this paper:1.Aiming at the problem of low communication efficiency in conventional federated learning scenarios,a global sparse adaptive aggregated stochastic gradient algorithm(The Global Sparse with Adaptive Aggregated Stochastic Gradients,GSASG)is proposed.Specifically,GSASG determines the clients that need to communicate with the parameter server according to the adaptive aggregation rules,and then sparsely uploads and downloads gradient information.The theoretical analysis and comparison with advanced algorithms show that GSASG further reduces the number of communication rounds,communication bits,and storage overhead of the client in the distributed system.Deep neural network training experiments show that GSASG can significantly reduce communication costs without sacrificing or even improving model performance.Taking the MNIST dataset as an example,in terms of the number of communication rounds,GSASG is 91%higher than sparse communication,90%higher than adaptive aggregation gradient,and 58%higher than sparse communication combined with adaptive aggregation gradient.In terms of communication bits,GSASG improves by 99%over previous algorithms.2.Aiming at the problem of low communication efficiency in large data set federated learning scenarios,the GSASG algorithm is extended,and a hierarchical global sparse adaptive aggregation stochastic gradient algorithm(The Hierarchical and Global Sparse with Adaptive Aggregated Stochastic Gradients,HGSASG)is proposed.Specifically,when a large batch of data sets is divided on the local client,first the gradient compression is performed on each divided data set locally on the client,and then all batches of compressed gradients are compressed locally on the client.local gradient compression,and finally complete a global gradient compression on the server.Experimental results show that HGSASG has good performance in dealing with large-scale data sets.Compared with GSASG,HGSASG can reduce the number of communication rounds and bits while further improving the model convergence performance.3.Aiming at the gradient leakage problem in federated learning,on the basis of GSASG,a global sparse adaptive aggregation stochastic gradient algorithm based on differential privacy(The Global Sparse with Adaptive Aggregated Stochastic Gradients based on Differential Privacy,GSASG-DP)is proposed.Specifically,firstly,the client compresses the local gradient,and then adds Gaussian noise to the gradient to achieve disturbance,so that the uploaded local training gradient is full of(ε,δ)-checking privacy,and a small number of sparse backbone Gradient perturbation can minimize the impact on the communication efficiency of federated learning and reduce the consumption of the privacy budget.Finally,global gradient compression and error accumulation are performed on the server to reduce the privacy loss caused by the aggregation of gradients with high dimensional plus noise.When the accuracy of the loss model is within an acceptable range,the advantage of GSASG-DP lies in the privacy protection of gradient information.