Research On Differential Privacy In Deep Learning

As current mainstream algorithm,deep learning has layer-by-layer abstract implementation principle which causes the training data of model to face great risk of privacy leakage.Differential privacy stands out among many privacy-preserving algorithms because of solid mathematical foundation.Differential privacy can effectively prevent attackers from obtaining training dataset privacy through model parameters by adding noise in the training process of deep learning model.However,the addition of noise will reduce the accuracy of the model.In order to achieve an effective trade-off between privacy and accuracy in deep learning model,this thesis conducts the following three studies.The Differential Privacy Adaptive and Momental Bound algorithm(DPAda Mod)is proposed in this thesis.Adding noise in the gradient to protect model privacy,and using adaptive learning rate to reduce the number of model training iterations and the loss of privacy.In addition,the proposed DPAda Mod algorithm utilizes the characteristics of momental bound to effectively adjust the learning rate,improves the robustness of the initial learning rate and memory length parameters,and alleviates the difficulty of parameter adjustment in complex scenarios combining differential privacy and deep learning.Simulations shows that,compared with the traditional differential privacy optimization algorithm,the DPAda Mod algorithm reduces the privacy loss by about 34% while ensuring the accuracy,which effectively balances the privacy and accuracy of the model.To further improve the accuracy of differential Privacy Optimization Algorithm,the Adaptive Gradient Clip Differential Privacy algorithm(AGCDP)is proposed in this thesis.Designing adaptive gradient clip method and adding noise according to the gradient clip threshold to protect the privacy of the deep learning model.The AGCDP algorithm improves the accuracy of deep learning model through reducing the addition of unnecessary noise in the training process.Simulations shows that,compared with the traditional algorithm,the AGCDP algorithm goes further in the case of ensuring privacy,and its accuracy is improved by about 3%,which better balances the privacy and accuracy of the model.In order to improve the applicability of the differential privacy protection algorithm in deep neural networks,the effect of weight normalization and initialization in the trade-off between privacy and accuracy of deep neural networks is investigated,and the Weight Normalization-Initialization-Differential Privacy algorithm(WN-INIT-DP)is proposed in this thesis.We focus on the deep residual neural networks,and propose to replace the batch normalization layer in the deep residual neural network with the weight normalization layer,and select the corresponding initialization method according to the improved neural network structure.By constructing a stable neural network structure and accelerating the model convergence speed,an effective trade-off between privacy and accuracy of the model is achieved.Simulations show that the WN-INIT-DP algorithm can effectively improve the accuracy and reduce the privacy loss of the deep learning model through the combination of weight normalization and initialization.This thesis studies the trade-off between privacy and accuracy in deep learning model,the DPAda Mod algorithm,AGCDP algorithm and WN-INIT-DP algorithm are proposed through combining differential privacy optimization algorithms with adaptive learning rates,adaptive gradient clip,weight normalization and initialization.Simulations show that three algorithms proposed in this thesis effectively trade off the privacy and accuracy of the model.