Abstract:Temporal difference (TD) learning with nonlinear function approximation (nonlinear TD learning for short) permits evaluating policies using neural networks, and is a core component in modern deep reinforcement learning. Though significant advances have been made to improve its effectiveness, little attention has been paid to the data privacy faced when applying it in real applications. To mitigate the privacy concerns in practical applications of nonlinear TD learning, in this paper, we consider preserving its privacy under the notion of differential privacy (DP). This problem is challenging since nonlinear TD learning is usually studied in the formulation of stochastic nonconvex-strongly-concave optimization to obtain finite-sample analysis, which requires simultaneously preserving privacy on both primal and dual sides. To this end, we adopt a single-timescale algorithm, which optimizes both sides using learning rates of the same order, to avoid unnecessary privacy costs. Further, we achieve a good trade-off between the privacy and utility guarantees by perturbing gradients on both sides using Gaussian noises with well-calibrated variances. Consequently, our algorithm achieves rigorous (ε,δ)-DP guarantee with the utility upper bounded by ~O((d/log(1/δ))1/8 (nε)1/4) where n is the trajectory length and d is the ambient dimension of the feature space. Extensive experiments conducted in OpenAI Gym validate the advantages of our algorithm.

Accelerated Gradient Temporal Difference Learning

An Emphatic Approach to the Problem of Off-policy Temporal-Difference Learning

Gradient Descent Temporal Difference-Difference Learning

Target-Based Temporal Difference Learning

New Versions of Gradient Temporal Difference Learning

Temporal Difference Learning as Gradient Splitting

Gradient Temporal Difference with Momentum: Stability and Convergence

Gradient compensation traces based temporal difference learning

Revisiting a Design Choice in Gradient Temporal Difference Learning

Historical Temporal Difference Learning: Some Initial Results

TIDBD: Adapting Temporal-difference Step-sizes Through Stochastic Meta-descent

Decentralized Adaptive TD $(\lambda)$ Learning with Linear Function Approximation: Nonasymptotic Analysis

A Non-asymptotic Analysis of Non-parametric Temporal-Difference Learning

Adaptive Step-Size for Online Temporal Difference Learning

On the Statistical Benefits of Temporal Difference Learning

Online Sparse Temporal Difference Learning Based on Nested Optimization and Regularized Dual Averaging.

Finite time analysis of temporal difference learning with linear function approximation: Tail averaging and regularisation

Almost Sure Convergence of Average Reward Temporal Difference Learning

Differentially Private Temporal Difference Learning with Stochastic Nonconvex-Strongly-Concave Optimization

Reanalysis of Variance Reduced Temporal Difference Learning

Effective Multi-step Temporal-Difference Learning for Non-Linear Function Approximation