Abstract:Cooperative decentralized deep learning relies on direct information exchange between communicating agents, each with access to a local dataset which should be kept private. The goal is for all agents to achieve consensus on model parameters after training. However, sharing parameters with untrustworthy neighboring agents could leak exploitable information about local datasets. To combat this, we introduce differentially private decentralized learning that secures each agent's local dataset during and after cooperative training. In our approach, we generalize Differentially Private Stochastic Gradient Descent (DP-SGD) -- a popular differentially private training method for centralized deep learning -- to practical subgradient- and ADMM-based decentralized learning methods. Our algorithms' differential privacy guarantee holds for arbitrary deep learning objective functions, and we analyze the convergence properties for strongly convex objective functions. We compare our algorithms against centrally trained models on standard classification tasks and evaluate the relationships between performance, privacy budget, graph connectivity, and degree of training data overlap among agents. We find that differentially private gradient tracking is resistant to performance degradation under sparse graphs and non-uniform data distributions. Furthermore, we show that it is possible to learn a model achieving high accuracies, within 3% of DP-SGD on MNIST under (1, 10^-5)-differential privacy and within 6% of DP-SGD on CIFAR-100 under (10, 10^-5)-differential privacy, without ever sharing raw data with other agents. Open source code can be found at: <a class="link-external link-https" href="https://github.com/jbayrooti/dp-dec-learning" rel="external noopener nofollow">this https URL</a>.

A general differentially private learning framework for decentralized data

A(DP)$^2$SGD: Asynchronous Decentralized Parallel Stochastic Gradient Descent with Differential Privacy

Differentially Private Decentralized Deep Learning with Consensus Algorithms

A(DP)$^2$2SGD: Asynchronous Decentralized Parallel Stochastic Gradient Descent with Differential Privacy

Generalizing Differentially Private Decentralized Deep Learning with Multi-Agent Consensus

From Noisy Fixed-Point Iterations to Private ADMM for Centralized and Federated Learning

Stochastic Differentially Private and Fair Learning

How to Democratise and Protect AI: Fair and Differentially Private Decentralised Deep Learning

Differentially Private Byzantine-Robust Federated Learning

Practical Differentially Private and Byzantine-resilient Federated Learning

Differentially Private Online Federated Learning with Correlated Noise

Mitigating Disparate Impact on Model Accuracy in Differentially Private Learning.

Differentially Private Federated Learning with an Adaptive Noise Mechanism

Locally Differentially Private Distributed Deep Learning via Knowledge Distillation

Accurate Differentially Private Deep Learning on the Edge

Adap DP-FL: Differentially Private Federated Learning with Adaptive Noise

Towards Differentially Private Text Representations

Locally Differentially Private Distributed Online Learning with Guaranteed Optimality

A Learning-based Declarative Privacy-Preserving Framework for Federated Data Management

Utility Optimization of Federated Learning with Differential Privacy

DPDR: Gradient Decomposition and Reconstruction for Differentially Private Deep Learning