Abstract:To embrace the era of big data, there has been growing interest in designing distributed machine learning to exploit the collective computing power of the local computing nodes. Alternating Direction Method of Multipliers (ADMM) is one of the most popular methods. This method applies iterative local computations over local datasets at each agent and computation results exchange between the neighbors. During this iterative process, data privacy leakage arises when performing local computation over sensitive data. Although many differentially private ADMM algorithms have been proposed to deal with such privacy leakage, they still have to face many challenging issues such as low model accuracy over strict privacy constraints and requiring strong assumptions of convexity of the objective function. To address those issues, in this paper, we propose a differentially private robust ADMM algorithm (PR-ADMM) with Gaussian mechanism. We employ two kinds of noise variance decay schemes to carefully adjust the noise addition in the iterative process and utilize a threshold to eliminate the too noisy results from neighbors. We also prove that PR-ADMM satisfies dynamic zero-concentrated differential privacy (dynamic zCDP) and a total privacy loss is given by (∈, δ)-differential privacy. From a theoretical point of view, we analyze the convergence rate of PR-ADMM for general convex objectives, which is O(1/K) with K being the number of iterations. The performance of the proposed algorithm is evaluated on real-world datasets. The experimental results show that the proposed algorithm outperforms other differentially private ADMM based algorithms under the same total privacy loss.

Distributed Logistic Regression with Differential Privacy

Distributed Online Bandit Linear Regressions with Differential Privacy.

Differential Privacy-preserving Distributed Machine Learning

Locally Differentially Private Distributed Online Learning with Guaranteed Optimality

Distributed Dynamic Online Learning with Differential Privacy Via Path-Length Measurement.

Differential Privacy for Sparse Classification Learning

Privacy-Preserving Multiparty Learning For Logistic Regression

Distributed Least-Squares Optimization Solvers with Differential Privacy

Transfer Learning for Logistic Regression with Differential Privacy

Guaranteed Distributed Machine Learning: Privacy-preserving Empirical Risk Minimization

Enabling Differentially Private in Big Data Machine Learning

Privacy-Preserving Distributed Optimization and Learning

Locally Differentially Private Gradient Tracking for Distributed Online Learning over Directed Graphs

Privacy-preserving multi-party logistic regression in cloud computing

Secure and Differentially Private Bayesian Learning on Distributed Data

Privacy Preserving Linear Regression Modeling of Distributed Databases

A Differential Privacy Preserving Approach for Logistic Regression in Genome-Wide Association Studies

Adaptive Distributed Differential Privacy with SGD

Privacy-Preserving Distributed Linear Regression on High-Dimensional Data

Distributed empirical risk minimization with differential privacy

Differentially Private Robust ADMM for Distributed Machine Learning.