Abstract:Unifying seemingly disparate algorithmic ideas to produce better performing algorithms has been a longstanding goal in reinforcement learning. As a primary example, TD($\lambda$) elegantly unifies one-step TD prediction with Monte Carlo methods through the use of eligibility traces and the trace-decay parameter $\lambda$. Currently, there are a multitude of algorithms that can be used to perform TD control, including Sarsa, $Q$-learning, and Expected Sarsa. These methods are often studied in the one-step case, but they can be extended across multiple time steps to achieve better performance. Each of these algorithms is seemingly distinct, and no one dominates the others for all problems. In this paper, we study a new multi-step action-value algorithm called $Q(\sigma)$ which unifies and generalizes these existing algorithms, while subsuming them as special cases. A new parameter, $\sigma$, is introduced to allow the degree of sampling performed by the algorithm at each step during its backup to be continuously varied, with Sarsa existing at one extreme (full sampling), and Expected Sarsa existing at the other (pure expectation). $Q(\sigma)$ is generally applicable to both on- and off-policy learning, but in this work we focus on experiments in the on-policy case. Our results show that an intermediate value of $\sigma$, which results in a mixture of the existing algorithms, performs better than either extreme. The mixture can also be varied dynamically which can result in even greater performance.

Multi-Step Truncated Q Learning Algorithm

A Multi-step Q Reinforcement Learning Algorithm

A Unified Approach for Multi-step Temporal-Difference Learning with Eligibility Traces in Reinforcement Learning

A Fuzzy Multi-Step Q Learning Algorithm Based On Q(Lambda)- Learning And Its Application

Multi-step backtrack Q-learning based dynamic optimal algorithm for auto generation control order dispatch

Multi-Step Q Learning Algorithm for Task Scheduling in Collaborative Design

Multi-step Reinforcement Learning: A Unifying Algorithm

An Improved Tentative Q Learning Algorithm for Robot Learning.

A Truncated Multi-step Actor--Critic Reinforcement Learning Algorithm

Metropolis Policy-based Multi-step Q Learning Algorithm and Performance Simulation

Two-Step Q-Learning

Understanding Multi-Step Deep Reinforcement Learning: A Systematic Study of the DQN Target

Reactive Power Optimization Calculation Based On Multi-Step Q(Lambda) Learning Algorithm

Multi-Swarm Cuckoo Search Algorithm with <i>Q-</i>Learning Model

Multi-Objective Optimal Power Flow Calculation Based on Multi-Step Q(λ) Learning Algorithm

Multi-Swarm Cuckoo Search Algorithm with Q-Learning Model

Multi-Agent Alternate Q-Learning.

A Multi-Agent Q-Learning with Value Function Approximation Based on Single-leader Multi-followers Stackelberg Game

Exploiting Multiple Question Factors for Knowledge Tracing

Multi-hop Knowledge Base Question Answering with an Iterative Sequence Matching Model

An Optimized Q-Learning Algorithm Based on the Thinking of Tabu Search