Abstract:In order to train agents that can quickly adapt to new objectives or reward functions, efficient unsupervised representation learning in sequential decision-making environments can be important. Frameworks such as the Exogenous Block Markov Decision Process (Ex-BMDP) have been proposed to formalize this representation-learning problem (Efroni et al., 2022b). In the Ex-BMDP framework, the agent's high-dimensional observations of the environment have two latent factors: a controllable factor, which evolves deterministically within a small state space according to the agent's actions, and an exogenous factor, which represents time-correlated noise, and can be highly complex. The goal of the representation learning problem is to learn an encoder that maps from observations into the controllable latent space, as well as the dynamics of this space. Efroni et al. (2022b) has shown that this is possible with a sample complexity that depends only on the size of the controllable latent space, and not on the size of the noise factor. However, this prior work has focused on the episodic setting, where the controllable latent state resets to a specific start state after a finite horizon. By contrast, if the agent can only interact with the environment in a single continuous trajectory, prior works have not established sample-complexity bounds. We propose STEEL, the first provably sample-efficient algorithm for learning the controllable dynamics of an Ex-BMDP from a single trajectory, in the function approximation setting. STEEL has a sample complexity that depends only on the sizes of the controllable latent space and the encoder function class, and (at worst linearly) on the mixing time of the exogenous noise factor. We prove that STEEL is correct and sample-efficient, and demonstrate STEEL on two toy problems. Code is available at: <a class="link-external link-https" href="https://github.com/midi-lab/steel" rel="external noopener nofollow">this https URL</a>.

Trajectory Sampling Value Iteration: Improved Dyna Search for MDPs

Dyna-H: a heuristic planning reinforcement learning algorithm applied to role-playing-game strategy decision systems

DynaSTI: Dynamics Modeling with Sequential Temporal Information for Reinforcement Learning in Atari

Adaptive trajectory-constrained exploration strategy for deep reinforcement learning

Improved Exploration through Latent Trajectory Optimization in Deep Deterministic Policy Gradient

A Heuristic Dyna Optimizing Algorithm Using Approximate Model Representation

Active Training Trajectory Generation for Inverse Dynamics Model Learning with Deep Neural Networks

Efficient Reinforcement Learning in Continuous State and Action Spaces with Dyna and Policy Approximation.

Deep Q-learning Sampling Based on Advantages

Optimizing Trajectories with Closed-Loop Dynamic SQP

Sampling Efficient Deep Reinforcement Learning through Preference-Guided Stochastic Exploration

Improving Trajectory Optimization using a Roadmap Framework

Improved Dyna-Q: A Reinforcement Learning Method Focused via Heuristic Graph for AGV Path Planning in Dynamic Environments

Trial and Error: Exploration-Based Trajectory Optimization of LLM Agents

Active Learning of Dynamics Using Prior Domain Knowledge in the Sampling Process

An Improved Dynamic Window Approach Based on Reinforcement Learning for the Trajectory Planning of Automated Guided Vehicles

Learning a Fast Mixing Exogenous Block MDP using a Single Trajectory

Deep Gaussian Covariance Network with Trajectory Sampling for Data-Efficient Policy Search

Trial and Error: Exploration-Based Trajectory Optimization for LLM Agents

Free from Bellman Completeness: Trajectory Stitching via Model-based Return-conditioned Supervised Learning