Abstract:One of the main obstacles to broad application of reinforcement learning methods is the parameter sensitivity of our core learning algorithms. In many large-scale applications, online computation and function approximation represent key strategies in scaling up reinforcement learning algorithms. In this setting, we have effective and reasonably well understood algorithms for adapting the learning-rate parameter, online during learning. Such meta-learning approaches can improve robustness of learning and enable specialization to current task, improving learning speed. For temporal-difference learning algorithms which we study here, there is yet another parameter, $\lambda$, that similarly impacts learning speed and stability in practice. Unfortunately, unlike the learning-rate parameter, $\lambda$ parametrizes the objective function that temporal-difference methods optimize. Different choices of $\lambda$ produce different fixed-point solutions, and thus adapting $\lambda$ online and characterizing the optimization is substantially more complex than adapting the learning-rate parameter. There are no meta-learning method for $\lambda$ that can achieve (1) incremental updating, (2) compatibility with function approximation, and (3) maintain stability of learning under both on and off-policy sampling. In this paper we contribute a novel objective function for optimizing $\lambda$ as a function of state rather than time. We derive a new incremental, linear complexity $\lambda$-adaption algorithm that does not require offline batch updating or access to a model of the world, and present a suite of experiments illustrating the practicality of our new algorithm in three different settings. Taken together, our contributions represent a concrete step towards black-box application of temporal-difference learning methods in real world problems.

Decentralized Adaptive TD $(\lambda)$ Learning with Linear Function Approximation: Nonasymptotic Analysis

Finite-Sample Analysis of Decentralized Temporal-Difference Learning with Linear Function Approximation

Finite-Time Analysis of Adaptive Temporal Difference Learning with Deep Neural Networks

Provable distributed adaptive temporal-difference learning over time-varying networks

Finite-Time Convergence Rates of Decentralized Local Markovian Stochastic Approximation

Exact Formulas for Finite-Time Estimation Errors of Decentralized Temporal Difference Learning with Linear Function Approximation

Decentralized TD(0) with Gradient Tracking

Byzantine-Resilient Decentralized Policy Evaluation with Linear Function Approximation

Finite-Time Bounds for AMSGrad-Enhanced Neural TD

An Emphatic Approach to the Problem of Off-policy Temporal-Difference Learning

Statistical Inference for Temporal Difference Learning with Linear Function Approximation

An Improved Finite-time Analysis of Temporal Difference Learning with Deep Neural Networks

Finite-Time Analysis of Temporal Difference Learning: Discrete-Time Linear System Perspective

Federated Temporal Difference Learning with Linear Function Approximation under Environmental Heterogeneity

Analysis of Off-Policy Multi-Step TD-Learning with Linear Function Approximation

A Greedy Approach to Adapting the Trace Parameter for Temporal Difference Learning

Almost Sure Convergence of Linear Temporal Difference Learning with Arbitrary Features

Target-Based Temporal Difference Learning

A Dynamically Adaptive Approach to Reducing Strategic Interference for Multi-agent Systems

Online Sparse Temporal Difference Learning Based on Nested Optimization and Regularized Dual Averaging.

An Empirical Evaluation of True Online TD(λ)