Abstract:Temporal difference (TD) learning with nonlinear function approximation (nonlinear TD learning for short) permits evaluating policies using neural networks, and is a core component in modern deep reinforcement learning. Though significant advances have been made to improve its effectiveness, little attention has been paid to the data privacy faced when applying it in real applications. To mitigate the privacy concerns in practical applications of nonlinear TD learning, in this paper, we consider preserving its privacy under the notion of differential privacy (DP). This problem is challenging since nonlinear TD learning is usually studied in the formulation of stochastic nonconvex-strongly-concave optimization to obtain finite-sample analysis, which requires simultaneously preserving privacy on both primal and dual sides. To this end, we adopt a single-timescale algorithm, which optimizes both sides using learning rates of the same order, to avoid unnecessary privacy costs. Further, we achieve a good trade-off between the privacy and utility guarantees by perturbing gradients on both sides using Gaussian noises with well-calibrated variances. Consequently, our algorithm achieves rigorous (ε,δ)-DP guarantee with the utility upper bounded by ~O((d/log(1/δ))1/8 (nε)1/4) where n is the trajectory length and d is the ambient dimension of the feature space. Extensive experiments conducted in OpenAI Gym validate the advantages of our algorithm.

Temporal Difference Learning as Gradient Splitting

Gradient Descent Temporal Difference-Difference Learning

Closing the gap between SVRG and TD-SVRG with Gradient Splitting

Statistical Inference for Temporal Difference Learning with Linear Function Approximation

An Emphatic Approach to the Problem of Off-policy Temporal-Difference Learning

Revisiting a Design Choice in Gradient Temporal Difference Learning

Accelerated Gradient Temporal Difference Learning

A Temporal-Difference Approach to Policy Gradient Estimation

Improved High-Probability Bounds for the Temporal Difference Learning Algorithm via Exponential Stability

Historical Temporal Difference Learning: Some Initial Results

New Versions of Gradient Temporal Difference Learning

A Non-asymptotic Analysis of Non-parametric Temporal-Difference Learning

On the Statistical Benefits of Temporal Difference Learning

Is Temporal Difference Learning Optimal? an Instance-Dependent Analysis

Finite time analysis of temporal difference learning with linear function approximation: Tail averaging and regularisation

A Convergent Off-Policy Temporal Difference Algorithm

Gradient Temporal Difference with Momentum: Stability and Convergence

Differentially Private Temporal Difference Learning with Stochastic Nonconvex-Strongly-Concave Optimization

Online Sparse Temporal Difference Learning Based on Nested Optimization and Regularized Dual Averaging.

Reanalysis of Variance Reduced Temporal Difference Learning

A Variance Minimization Approach to Temporal-Difference Learning