Abstract:Image classification is a fundamental artificial intelligence task that labels images into one of some predefined classes. However, training complex image classification models requires a large amount of computation resources and data in order to reach state-of-the-art performance. This demand drives the growth of distributed deep learning, where multiple agents cooperatively train global models with their individual datasets. Among such learning systems, decentralized learning is particularly attractive, as it can improve the efficiency and fault tolerance by eliminating the centralized parameter server, which could be the single point of failure or performance bottleneck. Although the agents do not need to disclose their training image samples, they exchange parameters with each other at each iteration, which can put them at the risk of data privacy leakage. Past works demonstrated the possibility of recovering training images from the exchanged parameters. One common defense direction is to adopt Differential Privacy (DP) to secure the optimization algorithms such as Stochastic Gradient Descent (SGD). Those DP-based methods mainly focus on standalone systems, or centralized distributed learning. How to enforce and optimize DP protection in decentralized learning systems is unknown and challenging, due to their complex communication topologies and distinct learning characteristics. In this paper, we design TOP- DP, a novel solution to optimize the differential privacy protection of decentralized image classification systems. The key insight of our solution is to leverage the unique features of decentralized communication topologies to reduce the noise scale and improve the model usability. (1) We enhance the DP-SGD algorithm with this topology-aware noise reduction strategy, and integrate the time-aware noise decay technique. (2) We design two novel learning protocols (synchronous and asynchronous) to protect systems with different network connectivities and topologies. We formally analyze and prove the DP requirement of our proposed solutions. Experimental evaluations demonstrate that our solution achieves a better trade-off between usability and privacy than prior works. To the best of our knowledge, this is the first DP optimization work from the perspective of network topologies.

Time-Efficient Ring-Topology Design for Decentralized Learning Via Device-to-Device Communications

A Ring Topology-based Communication-Efficient Scheme for D2D Wireless Federated Learning

Decentralized Edge Learning via Unreliable Device-to-Device Communications

Topology Learning for Heterogeneous Decentralized Federated Learning Over Unreliable D2D Networks

Communication-Efficient Topologies for Decentralized Learning with O(1) Consensus Rate

Efficient Collaborative Learning over Unreliable D2D Network: Adaptive Cluster Head Selection and Resource Allocation

Adaptive Cluster Head Selection and Spectrum Allocation for D2D-Enabled Collaborative Learning

Energy-Efficient Connectivity-Aware Learning Over Time-Varying D2D Networks

Hierarchical federated learning based on wireless D2D networks

Enhancing Decentralized and Personalized Federated Learning with Topology Construction

Topology Construction with Minimum Total Time for Geo-Distributed Decentralized Federated Learning.

Learning-Driven Decentralized Machine Learning in Resource-Constrained Wireless Edge Computing.

A fully decentralized distributed learning algorithm for latency communication networks

Topologies in distributed machine learning: Comprehensive survey, recommendations and future directions

Efficient Ring-topology Decentralized Federated Learning with Deep Generative Models for Industrial Artificial Intelligent

Topology-Aware Differential Privacy for Decentralized Image Classification

Communication-Efficient Topology Orchestration for Distributed Learning in UAV Networks

A Graph Neural Network Based Decentralized Learning Scheme

Aggregation Design and Resource Allocation for Decentralized Learning over Wireless Networks.

Towards Practical Overlay Networks for Decentralized Federated Learning