Abstract:To embrace the era of big data, there has been growing interest in designing distributed machine learning to exploit the collective computing power of the local computing nodes. Alternating Direction Method of Multipliers (ADMM) is one of the most popular methods. This method applies iterative local computations over local datasets at each agent and computation results exchange between the neighbors. During this iterative process, data privacy leakage arises when performing local computation over sensitive data. Although many differentially private ADMM algorithms have been proposed to deal with such privacy leakage, they still have to face many challenging issues such as low model accuracy over strict privacy constraints and requiring strong assumptions of convexity of the objective function. To address those issues, in this paper, we propose a differentially private robust ADMM algorithm (PR-ADMM) with Gaussian mechanism. We employ two kinds of noise variance decay schemes to carefully adjust the noise addition in the iterative process and utilize a threshold to eliminate the too noisy results from neighbors. We also prove that PR-ADMM satisfies dynamic zero-concentrated differential privacy (dynamic zCDP) and a total privacy loss is given by (∈, δ)-differential privacy. From a theoretical point of view, we analyze the convergence rate of PR-ADMM for general convex objectives, which is O(1/K) with K being the number of iterations. The performance of the proposed algorithm is evaluated on real-world datasets. The experimental results show that the proposed algorithm outperforms other differentially private ADMM based algorithms under the same total privacy loss.

Wireless Coded Distributed Learning with Gaussian-based Local Differential Privacy.

Green Differentially Private Coded Distributed Learning over Near-Field MIMO Systems

Joint Communication and Learning Design of Differential Privacy for Federated Learning over Multi-cell Networks.

Private and communication-efficient edge learning: a sparse differential gaussian-masking distributed SGD approach

A(DP)$^2$SGD: Asynchronous Decentralized Parallel Stochastic Gradient Descent with Differential Privacy

Privacy-Preserving Decentralized Inference with Graph Neural Networks in Wireless Networks

Differential Privacy and Distributed Online Learning for Wireless Big Data.

A(DP)$^2$2SGD: Asynchronous Decentralized Parallel Stochastic Gradient Descent with Differential Privacy

A New Noise Generating Method Based on Gaussian Sampling for Privacy Preservation

Layered Randomized Quantization for Communication-Efficient and Privacy-Preserving Distributed Learning

Privacy for Free: Wireless Federated Learning via Uncoded Transmission With Adaptive Power Control

Differentially Private Nash Equilibrium Seeking in Quadratic Network Games

Energy Efficient and Differentially Private Federated Learning via a Piggyback Approach

On differential privacy for federated learning in wireless systems with multiple base stations

Locally Differentially Private Gradient Tracking for Distributed Online Learning over Directed Graphs

Locally Differentially Private Distributed Online Learning with Guaranteed Optimality

Utility–Privacy Trade-Off in Distributed Machine Learning Systems

Differentially Private Robust ADMM for Distributed Machine Learning.

Matrix Gaussian Mechanisms for Differentially-Private Learning

Guaranteed Distributed Machine Learning: Privacy-preserving Empirical Risk Minimization

Distributed Least-Squares Optimization Solvers with Differential Privacy