Abstract: Off-policy reinforcement learning with eligibility traces is challenging because of the discrepancy between target policy and behavior policy. One common approach is to measure the difference between two policies in a probabilistic way, such as importance sampling and tree-backup. However, existing off-policy learning methods based on probabilistic policy measurement are inefficient when utilizing traces under a greedy target policy, which is ineffective for control problems. The traces are cut immediately when a non-greedy action is taken, which may lose the advantage of eligibility traces and slow down the learning process. Alternatively, some non-probabilistic measurement methods such as General Q($\lambda$) and Naive Q($\lambda$) never cut traces, but face convergence problems in practice. To address the above issues, this paper introduces a new method named TBQ($\sigma$), which effectively unifies the tree-backup algorithm and Naive Q($\lambda$). By introducing a new parameter $\sigma$ to illustrate the \emph{degree} of utilizing traces, TBQ($\sigma$) creates an effective integration of TB($\lambda$) and Naive Q($\lambda$) and continuous role shift between them. The contraction property of TB($\sigma$) is theoretically analyzed for both policy evaluation and control settings. We also derive the online version of TBQ($\sigma$) and give the convergence proof. We empirically show that, for $\epsilon\in(0,1]$ in $\epsilon$-greedy policies, there exists some degree of utilizing traces for $\lambda\in[0,1]$, which can improve the efficiency in trace utilization for off-policy reinforcement learning, to both accelerate the learning process and improve the performance.

Multi-step Reinforcement Learning: A Unifying Algorithm

A Unified Approach for Multi-step Temporal-Difference Learning with Eligibility Traces in Reinforcement Learning

Double Q($σ$) and Q($σ, λ$): Unifying Reinforcement Learning Control Algorithms

Gradient Q(σ, Λ): A Unified Algorithm with Function Approximation for Reinforcement Learning

Understanding Multi-Step Deep Reinforcement Learning: A Systematic Study of the DQN Target

Gradient Q : A Unified Algorithm with Function Approximation for Reinforcement Learning

A Multi-step Q Reinforcement Learning Algorithm

Context-aware Active Multi-Step Reinforcement Learning

TBQ($\sigma$): Improving Efficiency of Trace Utilization for Off-Policy Reinforcement Learning

Segmenting Action-Value Functions Over Time-Scales in SARSA using TD($Δ$)

A Dynamic Adjusting Reward Function Method for Deep Reinforcement Learning with Adjustable Parameters

Per-decision Multi-step Temporal Difference Learning with Control Variates

Time-Scale Separation in Q-Learning: Extending TD($\triangle$) for Action-Value Function Decomposition

Two-Step Q-Learning

A Deep Reinforcement Learning Architecture for Multi-stage Optimal Control

Multi-Timescale Ensemble Q-learning for Markov Decision Process Policy Optimization

Towards a Simple Approach to Multi-step Model-based Reinforcement Learning

Long N-step Surrogate Stage Reward to Reduce Variances of Deep Reinforcement Learning in Complex Problems

Provable and Practical: Efficient Exploration in Reinforcement Learning via Langevin Monte Carlo

Data Efficient Deep Reinforcement Learning with Action-Ranked Temporal Difference Learning

Simultaneous Double Q-learning with Conservative Advantage Learning for Actor-Critic Methods