Abstract:Online reinforcement learning is concerned with training an agent on-the-fly via dynamic interaction with the environment. Here, due to the specifics of the application, it is not generally possible to perform long pre-training, as it is commonly done in off-line, model-free approaches, which are akin to dynamic programming. Such applications may be found more frequently in industry, rather than in pure digital fields, such as cloud services, video games, database management, etc., where reinforcement learning has been demonstrating success. Online reinforcement learning, in contrast, is more akin to classical control, which utilizes some model knowledge about the environment. Stability of the closed-loop (agent plus the environment) is a major challenge for such online approaches. In this paper, we tackle this problem by a special fusion of online reinforcement learning with elements of classical control, namely, based on the Lyapunov theory of stability. The idea is to start the agent at once, without pre-training, and learn approximately optimal policy under specially designed constraints, which guarantee stability. The resulting approach was tested in an extensive experimental study with a mobile robot. A nominal parking controller was used as a baseline. It was observed that the suggested agent could always successfully park the robot, while significantly improving the cost. While many approaches may be exploited for mobile robot control, we suggest that the experiments showed the promising potential of online reinforcement learning agents based on Lyapunov-like constraints. The presented methodology may be utilized in safety-critical, industrial applications where stability is necessary.

Stabilizing reinforcement learning control: A modular framework for optimizing over all stable behavior

A modular framework for stabilizing deep reinforcement learning control

Stability-certified reinforcement learning: A control-theoretic perspective

A note on stabilizing reinforcement learning

Reinforcement Learning Controller Design for Affine Nonlinear Discrete-Time Systems Using Online Approximators

Actor-Critic Reinforcement Learning for Control With Stability Guarantee

Actively Learning Reinforcement Learning: A Stochastic Optimal Control Approach

Reinforcement Learning based Design of Linear Fixed Structure Controllers

Stable Modular Control via Contraction Theory for Reinforcement Learning

Online Reinforcement Learning-based Neural Network Controller Design for Affine Nonlinear Discrete-time Systems.

Feedback Linearization for Unknown Systems via Reinforcement Learning

Lyapunov-stable neural-network control

Lyapunov-stable Neural Control for State and Output Feedback: A Novel Formulation

Stochastic Reinforcement Learning with Stability Guarantees for Control of Unknown Nonlinear Systems

Stabilizing Neural Control Using Self-Learned Almost Lyapunov Critics

Reinforcement Learning Control of Constrained Dynamic Systems with Uniformly Ultimate Boundedness Stability Guarantee

Optimal PID and Antiwindup Control Design as a Reinforcement Learning Problem

A framework of learning controller with Lyapunov-based constraint and application

A framework for online, stabilizing reinforcement learning

Learning Stabilization Control from Observations by Learning Lyapunov-like Proxy Models

Solving Stabilize-Avoid Optimal Control via Epigraph Form and Deep Reinforcement Learning