Abstract:Image classification is a fundamental artificial intelligence task that labels images into one of some predefined classes. However, training complex image classification models requires a large amount of computation resources and data in order to reach state-of-the-art performance. This demand drives the growth of distributed deep learning, where multiple agents cooperatively train global models with their individual datasets. Among such learning systems, decentralized learning is particularly attractive, as it can improve the efficiency and fault tolerance by eliminating the centralized parameter server, which could be the single point of failure or performance bottleneck. Although the agents do not need to disclose their training image samples, they exchange parameters with each other at each iteration, which can put them at the risk of data privacy leakage. Past works demonstrated the possibility of recovering training images from the exchanged parameters. One common defense direction is to adopt Differential Privacy (DP) to secure the optimization algorithms such as Stochastic Gradient Descent (SGD). Those DP-based methods mainly focus on standalone systems, or centralized distributed learning. How to enforce and optimize DP protection in decentralized learning systems is unknown and challenging, due to their complex communication topologies and distinct learning characteristics. In this paper, we design TOP- DP, a novel solution to optimize the differential privacy protection of decentralized image classification systems. The key insight of our solution is to leverage the unique features of decentralized communication topologies to reduce the noise scale and improve the model usability. (1) We enhance the DP-SGD algorithm with this topology-aware noise reduction strategy, and integrate the time-aware noise decay technique. (2) We design two novel learning protocols (synchronous and asynchronous) to protect systems with different network connectivities and topologies. We formally analyze and prove the DP requirement of our proposed solutions. Experimental evaluations demonstrate that our solution achieves a better trade-off between usability and privacy than prior works. To the best of our knowledge, this is the first DP optimization work from the perspective of network topologies.

Zero redundancy distributed learning with differential privacy

Privacy-Preserving Collaborative Deep Learning with Unreliable Participants.

Distributed Learning for Large-Scale Models at Edge With Privacy Protection

DP-FP: Differentially Private Forward Propagation for Large Models

DPZero: Private Fine-Tuning of Language Models without Backpropagation

Towards Efficient and Scalable Training of Differentially Private Deep Learning

Private Fine-tuning of Large Language Models with Zeroth-order Optimization

Pre-training Differentially Private Models with Limited Public Data

Optimal Differentially Private Model Training with Public Data

A(DP)$^2$SGD: Asynchronous Decentralized Parallel Stochastic Gradient Descent with Differential Privacy

Equivariant Differentially Private Deep Learning: Why DP-SGD Needs Sparser Models

Differentially Private Optimization on Large Model at Small Cost

DPAdapter: Improving Differentially Private Deep Learning through Noise Tolerance Pre-training

Locally Differentially Private Distributed Deep Learning via Knowledge Distillation

Differentially Private Zeroth-Order Methods for Scalable Large Language Model Finetuning

Differentially Private Optimizers Can Learn Adversarially Robust Models

Locally Differentially Private Distributed Online Learning with Guaranteed Optimality

Improving Differentially Private Models with Active Learning

Topology-Aware Differential Privacy for Decentralized Image Classification

Toward Training at ImageNet Scale with Differential Privacy

Medical imaging deep learning with differential privacy