Abstract:Distributed online learning has been proven extremely effective in solving large-scale machine learning problems over streaming data. However, information sharing between learners in distributed learning also raises concerns about the potential leakage of individual learners' sensitive data. To mitigate this risk, differential privacy, which is widely regarded as the "gold standard" for privacy protection, has been widely employed in many existing results on distributed online learning. However, these results often face a fundamental tradeoff between learning accuracy and privacy. In this paper, we propose a locally differentially private gradient tracking based distributed online learning algorithm that successfully circumvents this tradeoff. We prove that the proposed algorithm converges in mean square to the exact optimal solution while ensuring rigorous local differential privacy, with the cumulative privacy budget guaranteed to be finite even when the number of iterations tends to infinity. The algorithm is applicable even when the communication graph among learners is directed. To the best of our knowledge, this is the first result that simultaneously ensures learning accuracy and rigorous local differential privacy in distributed online learning over directed graphs. We evaluate our algorithm's performance by using multiple benchmark machine-learning applications, including logistic regression of the "Mushrooms" dataset and CNN-based image classification of the "MNIST" and "CIFAR-10" datasets, respectively. The experimental results confirm that the proposed algorithm outperforms existing counterparts in both training and testing accuracies.

Differentially Private Distributed Online Learning over Time‐varying Digraphs Via Dual Averaging

Differentially Private Distributed Online Optimization in Time-varying Networks Via Dual Averaging Push

Privacy-Preserving Distributed Online Optimization Over Unbalanced Digraphs via Subgradient Rescaling

Differentially private distributed online optimization via push-sum one-point bandit dual averaging

Privacy-Preserving Decentralized Dual Averaging for Online Optimization Over Directed Networks

Distributed Dynamic Online Learning with Differential Privacy Via Path-Length Measurement.

Distributed Online Private Learning of Convex Nondecomposable Objectives

Locally Differentially Private Distributed Online Learning with Guaranteed Optimality

A(DP)$^2$SGD: Asynchronous Decentralized Parallel Stochastic Gradient Descent with Differential Privacy

A(DP)$^2$2SGD: Asynchronous Decentralized Parallel Stochastic Gradient Descent with Differential Privacy

Privacy-Preserving Dual Averaging with Arbitrary Initial Conditions for Distributed Optimization

Locally Differentially Private Gradient Tracking for Distributed Online Learning over Directed Graphs

Privacy-Protected Decentralized Dual Averaging Push with Edge-Based Correlated Perturbations over Time-Varying Directed Networks.

Distributed DP-Helmet: Scalable Differentially Private Non-interactive Averaging of Single Layers

Differentially Private Distributed Resource Allocation Via Deviation Tracking

Privacy-Preserving Distributed Online Stochastic Optimization with Time-Varying Distributions

Distributed Delayed Dual Averaging for Distributed Optimization over Time-Varying Digraphs

Differentially Private Distributed Online Mirror Descent Algorithm

Differentially Private Distributed Estimation and Learning

Differentially Private Temporal Difference Learning with Stochastic Nonconvex-Strongly-Concave Optimization

Differentially Private Distributed Stochastic Optimization with Time-Varying Sample Sizes