Online Nonstochastic Model-Free Reinforcement Learning

Udaya Ghai,Arushi Gupta,Wenhan Xia,Karan Singh,Elad Hazan
2023-11-01
Abstract:We investigate robust model-free reinforcement learning algorithms designed for environments that may be dynamic or even adversarial. Traditional state-based policies often struggle to accommodate the challenges imposed by the presence of unmodeled disturbances in such settings. Moreover, optimizing linear state-based policies pose an obstacle for efficient optimization, leading to nonconvex objectives, even in benign environments like linear dynamical systems. Drawing inspiration from recent advancements in model-based control, we introduce a novel class of policies centered on disturbance signals. We define several categories of these signals, which we term pseudo-disturbances, and develop corresponding policy classes based on them. We provide efficient and practical algorithms for optimizing these policies. Next, we examine the task of online adaptation of reinforcement learning agents in the face of adversarial disturbances. Our methods seamlessly integrate with any black-box model-free approach, yielding provable regret guarantees when dealing with linear dynamics. These regret guarantees unconditionally improve the best-known results for bandit linear control in having no dependence on the state-space dimension. We evaluate our method over various standard RL benchmarks and demonstrate improved robustness.
Machine Learning,Optimization and Control
What problem does this paper attempt to address?
The paper attempts to address the problem of designing robust model-free reinforcement learning algorithms in the presence of dynamic or adversarial environments. Traditional state-based policies perform poorly when dealing with unmodeled disturbances, and optimizing linear state policies leads to non-convex objective functions, even in simple linear dynamical systems. To this end, the authors draw inspiration from recent advances in model predictive control, introducing a new class of policies based on disturbance signals and defining several classes of pseudo-disturbances. They develop corresponding policy classes and propose methods for online adaptation of reinforcement learning agents, seamlessly integrating with any black-box model-free methods under adversarial disturbances, providing provable regret guarantees for linear dynamical systems. Specifically, the main contributions of the paper include: 1. Defining disturbance-based policies, which differ from traditional state-feedback-dependent reinforcement learning methods. 2. Proposing 3 new methods for estimating pseudo-disturbances in model-free reinforcement learning environments. 3. Developing a new algorithm, MF-GPC, that combines existing reinforcement learning methods with the pseudo-disturbance framework. 4. Empirically evaluating the methods in the OpenAI Gym environment, showing improved performance and robustness over baseline methods. 5. Proving that the proposed algorithm can achieve sublinear regret when dealing with linear dynamics, with regret bounds that outperform existing best results in terms of dependence on the state space dimension.