Abstract:Unifying seemingly disparate algorithmic ideas to produce better performing algorithms has been a longstanding goal in reinforcement learning. As a primary example, TD($\lambda$) elegantly unifies one-step TD prediction with Monte Carlo methods through the use of eligibility traces and the trace-decay parameter $\lambda$. Currently, there are a multitude of algorithms that can be used to perform TD control, including Sarsa, $Q$-learning, and Expected Sarsa. These methods are often studied in the one-step case, but they can be extended across multiple time steps to achieve better performance. Each of these algorithms is seemingly distinct, and no one dominates the others for all problems. In this paper, we study a new multi-step action-value algorithm called $Q(\sigma)$ which unifies and generalizes these existing algorithms, while subsuming them as special cases. A new parameter, $\sigma$, is introduced to allow the degree of sampling performed by the algorithm at each step during its backup to be continuously varied, with Sarsa existing at one extreme (full sampling), and Expected Sarsa existing at the other (pure expectation). $Q(\sigma)$ is generally applicable to both on- and off-policy learning, but in this work we focus on experiments in the on-policy case. Our results show that an intermediate value of $\sigma$, which results in a mixture of the existing algorithms, performs better than either extreme. The mixture can also be varied dynamically which can result in even greater performance.

A Multi-step Q Reinforcement Learning Algorithm

Multi-Step Truncated Q Learning Algorithm

A Unified Approach for Multi-step Temporal-Difference Learning with Eligibility Traces in Reinforcement Learning

Gradient Q(σ, Λ): A Unified Algorithm with Function Approximation for Reinforcement Learning

Gradient Q : A Unified Algorithm with Function Approximation for Reinforcement Learning

Multi-step Reinforcement Learning: A Unifying Algorithm

Metropolis Policy-based Multi-step Q Learning Algorithm and Performance Simulation

Multi-step backtrack Q-learning based dynamic optimal algorithm for auto generation control order dispatch

A Fuzzy Multi-Step Q Learning Algorithm Based On Q(Lambda)- Learning And Its Application

Understanding Multi-Step Deep Reinforcement Learning: A Systematic Study of the DQN Target

Multi-Objective Reinforcement Learning Algorithm For Mosdmp In Unknown Environment

Optimal Exploration Algorithm of Multi-Agent Reinforcement Learning Methods (Student Abstract)

Multi-Step Q Learning Algorithm for Task Scheduling in Collaborative Design

Multi-Swarm Cuckoo Search Algorithm with Q-Learning Model

Multi-Swarm Cuckoo Search Algorithm with <i>Q-</i>Learning Model

A Multi-Agent Q-Learning with Value Function Approximation Based on Single-leader Multi-followers Stackelberg Game

Q-CF Multi-Agent Reinforcement Learning for Resource Allocation Problems

Multi-agent cooperative reinforcement learning algorithm based on practical reasoning

Towards a Simple Approach to Multi-step Model-based Reinforcement Learning

Reactive Power Optimization Calculation Based On Multi-Step Q(Lambda) Learning Algorithm

Probabilistic Reward-Based Reinforcement Learning for Multi-Agent Pursuit and Evasion