Abstract: Off-policy reinforcement learning with eligibility traces is challenging because of the discrepancy between target policy and behavior policy. One common approach is to measure the difference between two policies in a probabilistic way, such as importance sampling and tree-backup. However, existing off-policy learning methods based on probabilistic policy measurement are inefficient when utilizing traces under a greedy target policy, which is ineffective for control problems. The traces are cut immediately when a non-greedy action is taken, which may lose the advantage of eligibility traces and slow down the learning process. Alternatively, some non-probabilistic measurement methods such as General Q($\lambda$) and Naive Q($\lambda$) never cut traces, but face convergence problems in practice. To address the above issues, this paper introduces a new method named TBQ($\sigma$), which effectively unifies the tree-backup algorithm and Naive Q($\lambda$). By introducing a new parameter $\sigma$ to illustrate the \emph{degree} of utilizing traces, TBQ($\sigma$) creates an effective integration of TB($\lambda$) and Naive Q($\lambda$) and continuous role shift between them. The contraction property of TB($\sigma$) is theoretically analyzed for both policy evaluation and control settings. We also derive the online version of TBQ($\sigma$) and give the convergence proof. We empirically show that, for $\epsilon\in(0,1]$ in $\epsilon$-greedy policies, there exists some degree of utilizing traces for $\lambda\in[0,1]$, which can improve the efficiency in trace utilization for off-policy reinforcement learning, to both accelerate the learning process and improve the performance.

A Unified Approach for Multi-step Temporal-Difference Learning with Eligibility Traces in Reinforcement Learning

Multi-step Reinforcement Learning: A Unifying Algorithm

Gradient Q(σ, Λ): A Unified Algorithm with Function Approximation for Reinforcement Learning

TBQ(σ): Improving Efficiency of Trace Utilization for Off-Policy Reinforcement Learning.

TBQ($\sigma$): Improving Efficiency of Trace Utilization for Off-Policy Reinforcement Learning

Gradient Q : A Unified Algorithm with Function Approximation for Reinforcement Learning

Double Q($σ$) and Q($σ, λ$): Unifying Reinforcement Learning Control Algorithms

Understanding Multi-Step Deep Reinforcement Learning: A Systematic Study of the DQN Target

META-Learning Eligibility Traces for More Sample Efficient Temporal Difference Learning

Parallel reinforcement learning with eligibility traces

A Multi-step Q Reinforcement Learning Algorithm

On Recursive Temporal Difference and Eligibility Traces

Multi-Step Truncated Q Learning Algorithm

Segmenting Action-Value Functions Over Time-Scales in SARSA using TD($Δ$)

A Dynamic Adjusting Reward Function Method for Deep Reinforcement Learning with Adjustable Parameters

Adaptive and Multiple Time-scale Eligibility Traces for Online Deep Reinforcement Learning

Off-policy Learning with Eligibility Traces: A Survey

One-shot learning and behavioral eligibility traces in sequential decision making

Per-decision Multi-step Temporal Difference Learning with Control Variates

Modified Retrace for Off-Policy Temporal Difference Learning.

Simplifying Deep Temporal Difference Learning