Abstract:To embrace the era of big data, there has been growing interest in designing distributed machine learning to exploit the collective computing power of the local computing nodes. Alternating Direction Method of Multipliers (ADMM) is one of the most popular methods. This method applies iterative local computations over local datasets at each agent and computation results exchange between the neighbors. During this iterative process, data privacy leakage arises when performing local computation over sensitive data. Although many differentially private ADMM algorithms have been proposed to deal with such privacy leakage, they still have to face many challenging issues such as low model accuracy over strict privacy constraints and requiring strong assumptions of convexity of the objective function. To address those issues, in this paper, we propose a differentially private robust ADMM algorithm (PR-ADMM) with Gaussian mechanism. We employ two kinds of noise variance decay schemes to carefully adjust the noise addition in the iterative process and utilize a threshold to eliminate the too noisy results from neighbors. We also prove that PR-ADMM satisfies dynamic zero-concentrated differential privacy (dynamic zCDP) and a total privacy loss is given by (∈, δ)-differential privacy. From a theoretical point of view, we analyze the convergence rate of PR-ADMM for general convex objectives, which is O(1/K) with K being the number of iterations. The performance of the proposed algorithm is evaluated on real-world datasets. The experimental results show that the proposed algorithm outperforms other differentially private ADMM based algorithms under the same total privacy loss.

Differentially private distributed optimization for multi-agent systems via the augmented Lagrangian algorithm

Differentially Private Distributed Resource Allocation Via Deviation Tracking

Differentially private distributed online optimization via push-sum one-point bandit dual averaging

Differentially Private and Communication-Efficient Distributed Nonconvex Optimization Algorithms

Distributed Least-Squares Optimization Solvers with Differential Privacy

Tailoring Gradient Methods for Differentially-Private Distributed Optimization

Differentially Private Nash Equilibrium Seeking in Quadratic Network Games

Customized Local Differential Privacy for Multi-Agent Distributed Optimization

Privacy-Preserving Distributed Online Optimization Over Unbalanced Digraphs via Subgradient Rescaling

Compressed Differentially Private Distributed Optimization with Linear Convergence

Differentially Private Average Consensus: Obstructions, Trade-Offs, and Optimal Algorithm Design

Differentially Private Dual Gradient Tracking for Distributed Resource Allocation

Privacy-Preserved Distributed Learning With Zeroth-Order Optimization

Differentially Private Distributed Convex Optimization via Functional Perturbation

Differentially Private Robust ADMM for Distributed Machine Learning.

Cryptography-Based Privacy-Preserving Method for Distributed Optimization over Time-Varying Directed Graphs with Enhanced Efficiency

Differentially Private Distributed Estimation and Learning

Privacy-Preserving Distributed ADMM With Event-Triggered Communication

ADMM Based Privacy-preserving Decentralized Optimization

Privacy-Preserving Push-Pull Method for Decentralized Optimization via State Decomposition

Differentially Private and Byzantine-Resilient Decentralized Nonconvex Optimization: System Modeling, Utility, Resilience, and Privacy Analysis