Differentially Private Machine Learning Approaches Via Margin Distribution Optimizing

With the huge progress of the Internet and big data,artificial intelligence,represented by machine learning,not only makes our lives more convenient but also threatens privacy.In this article,we study differential privacy,which has become a standard guarantee nowadays.Inspired by the recently proposed learning mechanism based on margin distribution optimizing,we design novel differentially private machine learning approaches.Our work has the following main contributions:(1)We propose the Margin Distribution Analysis(MDA)algorithm,which is suitable for differentially private learning.MDA breaks up its margin distribution formulation into strong convex functions,which can be solved by differentially private optimizers efficiently.It also relieves the class-imbalance problem by distinguishing and balancing the margin means of different classes.With theoretical analysis,we found MDA could maintain moderate margin distribution even serious class-imbalance happens.In experiments,MDA demonstrates many advantages,including the rapid velocity of convergence,good robustness,and parameter insensitivity.(2)We proposed the Private-Public Stochastic Gradient Descent(PPSGD)algorithm,which is used to solve the objective of margin distribution better.PPSGD utilizes a small amount of public data efficiently,it adjusts privacy budget and gradient clipping dynamically in differentially private stochastic gradient descent,and borrows the idea of model reuse to fine-tune its result further.With theoretical analysis,we prove that PPSGD is differentially private for non-public data.In experiments,PPSGD improves the performance of differentially private empirical risk minimization and outperforms other existing approaches.