Abstract:Value function approximation, such as Q-learning, is widely used in the discrete control rather than the continuous one because the optimal action in the discrete setting is more easily selected. Optimizing the action is a non-convex optimization problem with respect to the complex value function. Some notable studies simplify the non-convex optimization problem by assuming the value function as quadratic in the actions or by discretizing the action space. However, the performance of the output policy will decline if these studies’ premises do not hold. In order to address the problem, we propose a framework that combines swarm intelligence algorithms with value-based Reinforcement Learning, where the swarm intelligence algorithms are employed to search for the optimal action with respect to the state and the value function. To ensure the correctness of this framework, we conditionally claim the convergence rate of swarm intelligence algorithms with high probability. We then implement it by searching the batch optimal actions to various states on the GPU platform for the batch training. Furthermore, we employ the population-based atomic actions for the compatibility with the existing related work about solving discrete control problems. Four classical control models and four robot simulation environments are utilized in the comparisons. According to empirical results, our framework outputs a policy comparable with that of the policy-based algorithms by 10% timesteps in the continuous control. Note to Practitioners—This paper is motivated by the exploration-exploitation dilemma of Reinforcement Learning to solve continuous control tasks. To balance the exploration and exploitation, the stochastic exploration and the prioritized exploration are roughly two feasible ways, where the prioritized one is a better choice due to the higher data efficiency than the stochastic one, e.g. $varepsilon$ -greedy. Normally, the prioritized exploration works well in the value-based Reinforcement Learning algorithms rather than the policy-based ones; meanwhile, the policy-based algorithms are more suitable to continuous control tasks than the value-based ones. To tackle this conflict, we especially design a particle swarm optimization to maximize the Q-value of action in Q-learning. Our design can be hybridized by various swarm intelligence and value-based Reinforcement Learning algorithms. Also, it can be embedded in most intelligent control systems easily. The aim of this study is to solve the continuous control tasks by value-based algorithms as the first step of applying the prioritized exploration. The simulative results verify the effectiveness and efficiency of our design.

Model predictive control-based value estimation for efficient reinforcement learning

Model-Based Robot Learning Control with Uncertainty Directed Exploration

Model-Assisted Reinforcement Learning with Adaptive Ensemble Value Expansion

Deep Model-Based Reinforcement Learning for Predictive Control of Robotic Systems with Dense and Sparse Rewards

Model-based Reinforcement Learning with Multi-step Plan Value Estimation

Blending MPC & Value Function Approximation for Efficient Reinforcement Learning

An experimental study of two predictive reinforcement learning methods and comparison with model-predictive control

Data-Based Predictive Control Via Multistep Policy Gradient Reinforcement Learning

Data-Efficient Model-Based Reinforcement Learning For Robot Control

Diminishing Return of Value Expansion Methods in Model-Based Reinforcement Learning

A goal-conditioned policy search method with multi-timescale value function tuning

Continuous Control With Swarm Intelligence Based Value Function Approximation

Better Value Estimation in Q-learning-based Multi-Agent Reinforcement Learning

Value-Consistent Representation Learning for Data-Efficient Reinforcement Learning

Dynamic Horizon Value Estimation for Model-based Reinforcement Learning

Planning with Exploration: Addressing Dynamics Bottleneck in Model-based Reinforcement Learning

Plan to Predict: Learning an Uncertainty-Foreseeing Model for Model-Based Reinforcement Learning.

Model-Based Reinforcement Learning for Robotic Arm Control with Limited Environment Interaction

Controlling Estimation Error in Reinforcement Learning via Reinforced Operation

Mixed Reinforcement Learning for Efficient Policy Optimization in Stochastic Environments

Model Predictive Control of Quadruped Robot Based on Reinforcement Learning