Abstract:Distributed learning over data from sensor-based networks has been adopted to collaboratively train models on these sensitive data without privacy leakages. We present a distributed learning framework that involves the integration of secure multi-party computation and differential privacy. In our differential privacy method, we explore the potential of output perturbation and gradient perturbation and also progress with the cutting-edge methods of both techniques in the distributed learning domain. In our proposed multi-scheme output perturbation algorithm (MS-OP), data owners combine their local classifiers within a secure multi-party computation and later inject an appreciable amount of statistical noise into the model before they are revealed. In our Adaptive Iterative gradient perturbation (MS-GP) method, data providers collaboratively train a global model. During each iteration, the data owners aggregate their locally trained models within the secure multi-party domain. Since the conversion of differentially private algorithms are often naive, we improve on the method by a meticulous calibration of the privacy budget for each iteration. As the parameters of the model approach the optimal values, gradients are decreased and therefore require accurate measurement. We, therefore, add a fundamental line-search capability to enable our MS-GP algorithm to decide exactly when a more accurate measurement of the gradient is indispensable. Validation of our models on three (3) real-world datasets shows that our algorithm possesses a sustainable competitive advantage over the existing cutting-edge privacy-preserving requirements in the distributed setting.

Adaptive Distributed Differential Privacy with SGD

A(DP)$^2$SGD: Asynchronous Decentralized Parallel Stochastic Gradient Descent with Differential Privacy

A(DP)$^2$2SGD: Asynchronous Decentralized Parallel Stochastic Gradient Descent with Differential Privacy

Securing Distributed SGD Against Gradient Leakage Threats

Dynamic Differential-Privacy Preserving SGD

D-(DP)2SGD: Decentralized Parallel SGD with Differential Privacy in Dynamic Networks

Differentially Private SGD with Random Features

Differential Privacy-preserving Distributed Machine Learning

Locally Differentially Private Gradient Tracking for Distributed Online Learning over Directed Graphs

Locally Differentially Private Distributed Online Learning with Guaranteed Optimality

A Stochastic Gradient Descent Algorithm Based on Adaptive Differential Privacy

Scalable Differential Privacy Mechanisms for Real-Time Machine Learning Applications

Bolt-on Differential Privacy for Scalable Stochastic Gradient Descent-based Analytics

Guaranteed Distributed Machine Learning: Privacy-preserving Empirical Risk Minimization

Stochastic gradient descent with differentially private updates

Auditing Differentially Private Machine Learning: How Private is Private SGD?

Differentially private stochastic gradient descent with low-noise

An Improved Stochastic Gradient Descent Algorithm Based on Rényi Differential Privacy

Enhancing DP-SGD through Non-monotonous Adaptive Scaling Gradient Weight

Decentralized Parallel SGD with Privacy Preservation in Vehicular Networks