Abstract:Asynchronous Q-learning aims to learn the optimal action-value function (or Q-function) of a Markov decision process (MDP), based on a single trajectory of Markovian samples induced by a behavior policy. Focusing on a $\gamma $ -discounted MDP with state space $\mathcal {S}$ and action space $\mathcal {A}$ , we demonstrate that the $\ell _{\infty }$ -based sample complexity of classical asynchronous Q-learning — namely, the number of samples needed to yield an entrywise $\varepsilon $ -accurate estimate of the Q-function — is at most on the order of $\frac {1}{ \mu _{\mathsf {min}}(1-\gamma)^{5}\varepsilon ^{2}}+ \frac { t_{\mathsf {mix}}}{ \mu _{\mathsf {min}}(1-\gamma)}$ up to some logarithmic factor, provided that a proper constant learning rate is adopted. Here, $t_{\mathsf {mix}}$ and $\mu _{\mathsf {min}}$ denote respectively the mixing time and the minimum state-action occupancy probability of the sample trajectory. The first term of this bound matches the sample complexity in the synchronous case with independent samples drawn from the stationary distribution of the trajectory. The second term reflects the cost taken for the empirical distribution of the Markovian trajectory to reach a steady state, which is incurred at the very beginning and becomes amortized as the algorithm runs. Encouragingly, the above bound improves upon the state-of-the-art result by a factor of at least $|\mathcal {S}||\mathcal {A}|$ for all scenarios, and by a factor of at least $t_{\mathsf {mix}}|\mathcal {S}||\mathcal {A}|$ for any sufficiently small accuracy level $\varepsilon $ . Further, we demonstrate that the scaling on the effective horizon $\frac {1}{1-\gamma }$ can be improved by means of variance reduction.

Finite-Time Error Analysis of Online Model-Based Q-Learning with a Relaxed Sampling Model

Gradient Q(σ, Λ): A Unified Algorithm with Function Approximation for Reinforcement Learning

Gradient Q : A Unified Algorithm with Function Approximation for Reinforcement Learning

Finite-Time Analysis of Asynchronous Q-Learning Under Diminishing Step-Size From Control-Theoretic View

Sample Efficient Reinforcement Learning with Partial Dynamics Knowledge

Model-based Offline Reinforcement Learning with Lower Expectile Q-Learning

Settling the Sample Complexity of Model-Based Offline Reinforcement Learning

Finite-Time Error Analysis of Soft Q-Learning: Switching System Approach

Sample-Efficient Reinforcement Learning Via Conservative Model-Based Actor-Critic.

Finite-Time Analysis of Simultaneous Double Q-learning

On the Sample Complexity of Vanilla Model-Based Offline Reinforcement Learning with Dependent Samples

A Finite Sample Complexity Bound for Distributionally Robust Q-learning

Unified ODE Analysis of Smooth Q-Learning Algorithms

Minimax Optimal Q Learning with Nearest Neighbors

Model-Free Reinforcement Learning: from Clipped Pseudo-Regret to Sample Complexity

Sample Complexity of Variance-reduced Distributionally Robust Q-learning

Sample-Efficient Reinforcement Learning Is Feasible for Linearly Realizable MDPs with Limited Revisiting

Breaking the sample complexity barrier to regret-optimal model-free reinforcement learning

Actions Speak What You Want: Provably Sample-Efficient Reinforcement Learning of the Quantal Stackelberg Equilibrium from Strategic Feedbacks

Sample Complexity of Asynchronous Q-Learning: Sharper Analysis and Variance Reduction

Sample-efficient Safe Learning for Online Nonlinear Control with Control Barrier Functions