Abstract:This article presents a reinforcement learning framework for continuous-time dynamical systems without a priori discretization of time, state, and action. Based on the Hamilton-Jacobi-Bellman (HJB) equation for infinite-horizon, discounted reward problems, we derive algorithms for estimating value functions and improving policies with the use of function approximators. The process of value function estimation is formulated as the minimization of a continuous-time form of the temporal difference (TD) error. Update methods based on backward Euler approximation and exponential eligibility traces are derived, and their correspondences with the conventional residual gradient, TD(0), and TD(lambda) algorithms are shown. For policy improvement, two methods-a continuous actor-critic method and a value-gradient-based greedy policy-are formulated. As a special case of the latter, a nonlinear feedback control law using the value gradient and the model of the input gain is derived. The advantage updating, a model-free algorithm derived previously, is also formulated in the HJB-based framework. The performance of the proposed algorithms is first tested in a nonlinear control task of swinging a pendulum up with limited torque. It is shown in the simulations that (1) the task is accomplished by the continuous actor-critic method in a number of trials several times fewer than by the conventional discrete actor-critic method; (2) among the continuous policy update methods, the value-gradient-based policy with a known or learned dynamic model performs several times better than the actor-critic method; and (3) a value function update using exponential eligibility traces is more efficient and stable than that based on Euler approximation. The algorithms are then tested in a higher-dimensional task: cart-pole swing-up. This task is accomplished in several hundred trials using the value-gradient-based policy with a learned dynamic model.

Reinforcement Learning Method For Continuous State Space Based On Dynamic Neural Network

Reinforcement Learning Control for Nonlinear Systems Based on Elman Neural Network

Learning biped locomotion based on Q-learning and neural networks

A Q-learning Method for Continuous Space Based on Self-organizing Fuzzy RBF Network

A kind of weighted Q-learning for continuous state and action spaces

A PROPOSAL OF WEIGHTED Q-LEARNING FOR CONTINUOUS STATE AND ACTION SPACES

Q-Learning for Continuous State Space Based on a Support Vector Machine

Application of Adaptive Reinforcement Learning for State Space Construction

Parametrized deep q-networks learning: Reinforcement learning with discrete-continuous hybrid action space

A Priori Estimates for Deep Residual Network in Continuous-time Reinforcement Learning

Reinforcement learning in continuous time and space

Human-in-the-Loop Reinforcement Learning in Continuous-Action Space

Reinforcement Learning for Continuous Spaces Based on Gaussian Process Classifier

Reinforcement Learning Method Based On Semi-Parametric Regression Model

Continuous deep q-learning with model-based acceleration

Representation learning for continuous action spaces is beneficial for efficient policy learning

Temporal difference learning with incremental nearest neighbors in continuous spaces

An efficient reinforcement learning algorithm for continuous actions

A Gradient Algorithm for Neural-Network-Based Reinforcement Learning

Reinforcement Learning Method of Continuous State Adaptively Discretized Based on K-means Clustering