Abstract:Representation learning is a powerful tool that enables learning over large multitudes of agents or domains by enforcing that all agents operate on a shared set of learned features. However, many robotics or controls applications that would benefit from collaboration operate in settings with changing environments and goals, whereas most guarantees for representation learning are stated for static settings. Toward rigorously establishing the benefit of representation learning in dynamic settings, we analyze the regret of multi-task representation learning for linear-quadratic control. This setting introduces unique challenges. Firstly, we must account for and balance the $\textit{misspecification}$ introduced by an approximate representation. Secondly, we cannot rely on the parameter update schemes of single-task online LQR, for which least-squares often suffices, and must devise a novel scheme to ensure sufficient improvement. We demonstrate that for settings where exploration is "benign", the regret of any agent after $T$ timesteps scales as $\tilde O(\sqrt{T/H})$, where $H$ is the number of agents. In settings with "difficult" exploration, the regret scales as $\tilde O(\sqrt{d_u d_\theta} \sqrt{T} + T^{3/4}/H^{1/5})$, where $d_x$ is the state-space dimension, $d_u$ is the input dimension, and $d_\theta$ is the task-specific parameter count. In both cases, by comparing to the minimax single-task regret $O(\sqrt{d_x d_u^2}\sqrt{T})$, we see a benefit of a large number of agents. Notably, in the difficult exploration case, by sharing a representation across tasks, the effective task-specific parameter count can often be small $d_\theta < d_x d_u$. Lastly, we provide numerical validation of the trends we predict.

Towards Learning Controllable Representations of Physical Systems

Evaluation of state representation methods in robot hand-eye coordination learning from demonstration

Learning Controllable Elements Oriented Representations for Reinforcement Learning

Learning telic-controllable state representations

How Are Learned Perception-Based Controllers Impacted by the Limits of Robust Control?

Effective Representation Learning is More Effective in Reinforcement Learning Than You Think

Policy-Independent Behavioral Metric-Based Representation for Deep Reinforcement Learning

Towards Principled Representation Learning from Videos for Reinforcement Learning

Learning Invariant Representations for Reinforcement Learning without Reconstruction

Bootstrapped Representations in Reinforcement Learning

Learning Action-based Representations Using Invariance

Task-Induced Representation Learning

Bridging State and History Representations: Understanding Self-Predictive RL

Representation-Driven Reinforcement Learning

State Representations as Incentives for Reinforcement Learning Agents: A Sim2Real Analysis on Robotic Grasping

Can Direct Latent Model Learning Solve Linear Quadratic Gaussian Control?

Actively Learning Reinforcement Learning: A Stochastic Optimal Control Approach

Regret Analysis of Multi-task Representation Learning for Linear-Quadratic Adaptive Control

Value-Consistent Representation Learning for Data-Efficient Reinforcement Learning

Multi-Horizon Representations with Hierarchical Forward Models for Reinforcement Learning

Constrained reinforcement learning with statewise projection: a control barrier function approach