Investigating Reinforcement Learning Agents for Continuous State Space Environments

David Von Dollen

DOI: https://doi.org/10.48550/arXiv.1708.02378

2019-03-12

Abstract:Given an environment with continuous state spaces and discrete actions, we investigate using a Double Deep Q-learning Reinforcement Agent to find optimal policies using the LunarLander-v2 OpenAI gym environment.

Artificial Intelligence

What problem does this paper attempt to address?

Continuous control with deep reinforcement learning

Daan Wierstra,David Silver,Jonathan J. Hunt,N. Heess,T. Lillicrap,Yuval Tassa,A. Pritzel,Tom Erez

2015-09-09

Abstract:We adapt the ideas underlying the success of Deep Q-Learning to the continuous action domain. We present an actor-critic, model-free algorithm based on the deterministic policy gradient that can operate over continuous action spaces. Using the same learning algorithm, network architecture and hyper-parameters, our algorithm robustly solves more than 20 simulated physics tasks, including classic problems such as cartpole swing-up, dexterous manipulation, legged locomotion and car driving. Our algorithm is able to find policies whose performance is competitive with those found by a planning algorithm with full access to the dynamics of the domain and its derivatives. We further demonstrate that for many of the tasks the algorithm can learn policies end-to-end: directly from raw pixel inputs.

Computer Science,Mathematics
Investigating the Impact of Choice on Deep Reinforcement Learning for Space Controls

Nathaniel Hamilton,Kyle Dunlap,Kerianne L. Hobbs

2024-05-21

Abstract:For many space applications, traditional control methods are often used during operation. However, as the number of space assets continues to grow, autonomous operation can enable rapid development of control methods for different space related tasks. One method of developing autonomous control is Reinforcement Learning (RL), which has become increasingly popular after demonstrating promising performance and success across many complex tasks. While it is common for RL agents to learn bounded continuous control values, this may not be realistic or practical for many space tasks that traditionally prefer an on/off approach for control. This paper analyzes using discrete action spaces, where the agent must choose from a predefined list of actions. The experiments explore how the number of choices provided to the agents affects their measured performance during and after training. This analysis is conducted for an inspection task, where the agent must circumnavigate an object to inspect points on its surface, and a docking task, where the agent must move into proximity of another spacecraft and "dock" with a low relative speed. A common objective of both tasks, and most space tasks in general, is to minimize fuel usage, which motivates the agent to regularly choose an action that uses no fuel. Our results show that a limited number of discrete choices leads to optimal performance for the inspection task, while continuous control leads to optimal performance for the docking task.

Machine Learning,Systems and Control
Learning Agents With Prioritization and Parameter Noise in Continuous State and Action Space

Rajesh Mangannavar,Gopalakrishnan Srinivasaraghavan

DOI: https://doi.org/10.1007/978-3-030-22796-8_22

2024-10-15

Abstract:Among the many variants of RL, an important class of problems is where the state and action spaces are continuous -- autonomous robots, autonomous vehicles, optimal control are all examples of such problems that can lend themselves naturally to reinforcement based algorithms, and have continuous state and action spaces. In this paper, we introduce a prioritized form of a combination of state-of-the-art approaches such as Deep Q-learning (DQN) and Deep Deterministic Policy Gradient (DDPG) to outperform the earlier results for continuous state and action space problems. Our experiments also involve the use of parameter noise during training resulting in more robust deep RL models outperforming the earlier results significantly. We believe these results are a valuable addition for continuous state and action space problems.

Artificial Intelligence,Machine Learning
Addressing High-dimensional Continuous Action Space via Decomposed Discrete Policy-Critic

Yechen Zhang,Jian Sun,Gang Wang,Jie Chen

2023-01-01

Abstract:Reinforcement learning (RL) methods for discrete action spaces like DQNs are being widely used in tasks such as Atari games. However, they encounter difficulties when addressing continuous control tasks, since discretizing continuous action space incurs the curse-of-dimensionality. To tackle continuous control tasks via discretized actions, we propose a decomposed discrete policy-critic (D2PC) architecture, which was inspired by multi-agent RL (MARL) and associates with each action dimension a discrete policy, while leveraging a single critic network to provide a shared evaluation. Building on D2PC, we advocate soft stochastic D2PC (SD2PC) and deterministic D2PC (D3PC) methods with a discrete stochastic or deterministic policy, which show comparable or superior training performances relative to even continuous actor-critic methods. Additionally, we design a mechanism that allows D3PC to interact with continuous actor-critic methods, contributing to the Q-policy-critic (QPC) algorithm, which inherits the training efficiency of discrete RL and the near-optimal final performance of continuous RL algorithms. Substantial experimental results on several continuous benchmark tasks validate our claims.
Reinforcement Learning for Improving Agent Design

David Ha

DOI: https://doi.org/10.1162/artl_a_00301

2019-12-02

Abstract:In many reinforcement learning tasks, the goal is to learn a policy to manipulate an agent, whose design is fixed, to maximize some notion of cumulative reward. The design of the agent's physical structure is rarely optimized for the task at hand. In this work, we explore the possibility of learning a version of the agent's design that is better suited for its task, jointly with the policy. We propose an alteration to the popular OpenAI Gym framework, where we parameterize parts of an environment, and allow an agent to jointly learn to modify these environment parameters along with its policy. We demonstrate that an agent can learn a better structure of its body that is not only better suited for the task, but also facilitates policy learning. Joint learning of policy and structure may even uncover design principles that are useful for assisted-design applications. Videos of results at <a class="link-external link-https" href="https://designrl.github.io/" rel="external noopener nofollow">this https URL</a>

Machine Learning
Double A3C: Deep Reinforcement Learning on OpenAI Gym Games

Yangxin Zhong,Jiajie He,Lingjie Kong

2023-03-04

Abstract:Reinforcement Learning (RL) is an area of machine learning figuring out how agents take actions in an unknown environment to maximize its rewards. Unlike classical Markov Decision Process (MDP) in which agent has full knowledge of its state, rewards, and transitional probability, reinforcement learning utilizes exploration and exploitation for the model uncertainty. Under the condition that the model usually has a large state space, a neural network (NN) can be used to correlate its input state to its output actions to maximize the agent's rewards. However, building and training an efficient neural network is challenging. Inspired by Double Q-learning and Asynchronous Advantage Actor-Critic (A3C) algorithm, we will propose and implement an improved version of Double A3C algorithm which utilizing the strength of both algorithms to play OpenAI Gym Atari 2600 games to beat its benchmarks for our project.

Artificial Intelligence,Machine Learning
Deep Reinforcement Learning with Embedded LQR Controllers

Wouter Caarls

DOI: https://doi.org/10.1016/j.ifacol.2020.12.2261

2021-01-19

Abstract:Reinforcement learning is a model-free optimal control method that optimizes a control policy through direct interaction with the environment. For reaching tasks that end in regulation, popular discrete-action methods are not well suited due to chattering in the goal state. We compare three different ways to solve this problem through combining reinforcement learning with classical LQR control. In particular, we introduce a method that integrates LQR control into the action set, allowing generalization and avoiding fixing the computed control in the replay memory if it is based on learned dynamics. We also embed LQR control into a continuous-action method. In all cases, we show that adding LQR control can improve performance, although the effect is more profound if it can be used to augment a discrete action set.

Robotics,Machine Learning,Systems and Control
A further exploration of deep Multi-Agent Reinforcement Learning with Hybrid Action Space

Hongzhi Hua,Guixuan Wen,Kaigui Wu

DOI: https://doi.org/10.48550/arXiv.2208.14447

2022-08-30

Abstract:The research of extending deep reinforcement learning (drl) to multi-agent field has solved many complicated problems and made great achievements. However, almost all these studies only focus on discrete or continuous action space and there are few works having ever used multi-agent deep reinforcement learning to real-world environment problems which mostly have a hybrid action space. Therefore, in this paper, we propose two algorithms: deep multi-agent hybrid soft actor-critic (MAHSAC) and multi-agent hybrid deep deterministic policy gradients (MAHDDPG) to fill this gap. This two algorithms follow the centralized training and decentralized execution (CTDE) paradigm and could handle hybrid action space problems. Our experiences are running on multi-agent particle environment which is an easy multi-agent particle world, along with some basic simulated physics. The experimental results show that these algorithms have good performances.

Machine Learning,Artificial Intelligence,Multiagent Systems
DQN with model-based exploration: efficient learning on environments with sparse rewards

Stephen Zhen Gou,Yuyang Liu

DOI: https://doi.org/10.48550/arXiv.1903.09295

2019-03-22

Abstract:We propose Deep Q-Networks (DQN) with model-based exploration, an algorithm combining both model-free and model-based approaches that explores better and learns environments with sparse rewards more efficiently. DQN is a general-purpose, model-free algorithm and has been proven to perform well in a variety of tasks including Atari 2600 games since it's first proposed by Minh et el. However, like many other reinforcement learning (RL) algorithms, DQN suffers from poor sample efficiency when rewards are sparse in an environment. As a result, most of the transitions stored in the replay memory have no informative reward signal, and provide limited value to the convergence and training of the Q-Network. However, one insight is that these transitions can be used to learn the dynamics of the environment as a supervised learning problem. The transitions also provide information of the distribution of visited states. Our algorithm utilizes these two observations to perform a one-step planning during exploration to pick an action that leads to states least likely to be seen, thus improving the performance of exploration. We demonstrate our agent's performance in two classic environments with sparse rewards in OpenAI gym: Mountain Car and Lunar Lander.

Machine Learning
Dynamic Path Planning of Unknown Environment Based on Deep Reinforcement Learning

Xiaoyun Lei,Zhian Zhang,Peifang Dong

DOI: https://doi.org/10.1155/2018/5781591

2018-09-18

Journal of Robotics

Abstract:Dynamic path planning of unknown environment has always been a challenge for mobile robots. In this paper, we apply double Q-network (DDQN) deep reinforcement learning proposed by DeepMind in 2016 to dynamic path planning of unknown environment. The reward and punishment function and the training method are designed for the instability of the training stage and the sparsity of the environment state space. In different training stages, we dynamically adjust the starting position and target position. With the updating of neural network and the increase of greedy rule probability, the local space searched by agent is expanded. Pygame module in PYTHON is used to establish dynamic environments. Considering lidar signal and local target position as the inputs, convolutional neural networks (CNNs) are used to generalize the environmental state. Q-learning algorithm enhances the ability of the dynamic obstacle avoidance and local planning of the agents in environment. The results show that, after training in different dynamic environments and testing in a new environment, the agent is able to reach the local target position successfully in unknown dynamic environment.
Continuous Control of an Underground Loader Using Deep Reinforcement Learning

Sofi Backman,Daniel Lindmark,Kenneth Bodin,Martin Servin,Joakim Mörk,Håkan Löfgren

DOI: https://doi.org/10.3390/machines9100216

IF: 2.899

2021-09-27

Machines

Abstract:The reinforcement learning control of an underground loader was investigated in a simulated environment by using a multi-agent deep neural network approach. At the start of each loading cycle, one agent selects the dig position from a depth camera image of a pile of fragmented rock. A second agent is responsible for continuous control of the vehicle, with the goal of filling the bucket at the selected loading point while avoiding collisions, getting stuck, or losing ground traction. This relies on motion and force sensors, as well as on a camera and lidar. Using a soft actor–critic algorithm, the agents learn policies for efficient bucket filling over many subsequent loading cycles, with a clear ability to adapt to the changing environment. The best results—on average, 75% of the max capacity—were obtained when including a penalty for energy usage in the reward.

engineering, mechanical, electrical & electronic
Exploring Exploration: Comparing Children with RL Agents in Unified Environments

Eliza Kosoy,Jasmine Collins,David M. Chan,Sandy Huang,Deepak Pathak,Pulkit Agrawal,John Canny,Alison Gopnik,Jessica B. Hamrick

DOI: https://doi.org/10.48550/arXiv.2005.02880

IF: 14.4

2020-05-06

Artificial Intelligence

Abstract:Research in developmental psychology consistently shows that children explore the world thoroughly and efficiently and that this exploration allows them to learn. In turn, this early learning supports more robust generalization and intelligent behavior later in life. While much work has gone into developing methods for exploration in machine learning, artificial agents have not yet reached the high standard set by their human counterparts. In this work we propose using DeepMind Lab (Beattie et al., 2016) as a platform to directly compare child and agent behaviors and to develop new exploration techniques. We outline two ongoing experiments to demonstrate the effectiveness of a direct comparison, and outline a number of open research questions that we believe can be tested using this methodology.
Continual Reinforcement Learning in 3D Non-stationary Environments

Vincenzo Lomonaco,Karan Desai,Eugenio Culurciello,Davide Maltoni

DOI: https://doi.org/10.48550/arXiv.1905.10112

2020-04-21

Abstract:High-dimensional always-changing environments constitute a hard challenge for current reinforcement learning techniques. Artificial agents, nowadays, are often trained off-line in very static and controlled conditions in simulation such that training observations can be thought as sampled i.i.d. from the entire observations space. However, in real world settings, the environment is often non-stationary and subject to unpredictable, frequent changes. In this paper we propose and openly release CRLMaze, a new benchmark for learning continually through reinforcement in a complex 3D non-stationary task based on ViZDoom and subject to several environmental changes. Then, we introduce an end-to-end model-free continual reinforcement learning strategy showing competitive results with respect to four different baselines and not requiring any access to additional supervised signals, previously encountered environmental conditions or observations.

Machine Learning,Computer Vision and Pattern Recognition
Deep Reinforcement Learning in Parameterized Action Space

Matthew Hausknecht,Peter Stone

2024-05-03

Abstract:Recent work has shown that deep neural networks are capable of approximating both value functions and policies in reinforcement learning domains featuring continuous state and action spaces. However, to the best of our knowledge no previous work has succeeded at using deep neural networks in structured (parameterized) continuous action spaces. To fill this gap, this paper focuses on learning within the domain of simulated RoboCup soccer, which features a small set of discrete action types, each of which is parameterized with continuous variables. The best learned agent can score goals more reliably than the 2012 RoboCup champion agent. As such, this paper represents a successful extension of deep reinforcement learning to the class of parameterized action space MDPs.

Artificial Intelligence,Machine Learning,Multiagent Systems,Neural and Evolutionary Computing
Action Mapping for Reinforcement Learning in Continuous Environments with Constraints

Mirco Theile,Lukas Dirnberger,Raphael Trumpp,Marco Caccamo,Alberto L. Sangiovanni-Vincentelli

2024-12-06

Abstract:Deep reinforcement learning (DRL) has had success across various domains, but applying it to environments with constraints remains challenging due to poor sample efficiency and slow convergence. Recent literature explored incorporating model knowledge to mitigate these problems, particularly through the use of models that assess the feasibility of proposed actions. However, integrating feasibility models efficiently into DRL pipelines in environments with continuous action spaces is non-trivial. We propose a novel DRL training strategy utilizing action mapping that leverages feasibility models to streamline the learning process. By decoupling the learning of feasible actions from policy optimization, action mapping allows DRL agents to focus on selecting the optimal action from a reduced feasible action set. We demonstrate through experiments that action mapping significantly improves training performance in constrained environments with continuous action spaces, especially with imperfect feasibility models.

Machine Learning,Artificial Intelligence,Systems and Control
Pathfinding in Random Partially Observable Environments with Vision-Informed Deep Reinforcement Learning

Anthony Dowling

DOI: https://doi.org/10.48550/arXiv.2209.04801

2022-09-11

Abstract:Deep reinforcement learning is a technique for solving problems in a variety of environments, ranging from Atari video games to stock trading. This method leverages deep neural network models to make decisions based on observations of a given environment with the goal of maximizing a reward function that can incorporate cost and rewards for reaching goals. With the aim of pathfinding, reward conditions can include reaching a specified target area along with costs for movement. In this work, multiple Deep Q-Network (DQN) agents are trained to operate in a partially observable environment with the goal of reaching a target zone in minimal travel time. The agent operates based on a visual representation of its surroundings, and thus has a restricted capability to observe the environment. A comparison between DQN, DQN-GRU, and DQN-LSTM is performed to examine each models capabilities with two different types of input. Through this evaluation, it is been shown that with equivalent training and analogous model architectures, a DQN model is able to outperform its recurrent counterparts.

Machine Learning,Computer Vision and Pattern Recognition
Architecting and Visualizing Deep Reinforcement Learning Models

Alexander Neuwirth,Derek Riley

DOI: https://doi.org/10.48550/arXiv.2112.01451

IF: 14.4

2021-12-02

Artificial Intelligence

Abstract:To meet the growing interest in Deep Reinforcement Learning (DRL), we sought to construct a DRL-driven Atari Pong agent and accompanying visualization tool. Existing approaches do not support the flexibility required to create an interactive exhibit with easily-configurable physics and a human-controlled player. Therefore, we constructed a new Pong game environment, discovered and addressed a number of unique data deficiencies that arise when applying DRL to a new environment, architected and tuned a policy gradient based DRL model, developed a real-time network visualization, and combined these elements into an interactive display to help build intuition and awareness of the mechanics of DRL inference.
Human-in-the-Loop Reinforcement Learning in Continuous-Action Space

Biao Luo,Zhengke Wu,Fei Zhou,Bing-Chuan Wang

DOI: https://doi.org/10.1109/tnnls.2023.3289315

IF: 14.255

2023-01-01

IEEE Transactions on Neural Networks and Learning Systems

Abstract:Human-in-the-loop for reinforcement learning (RL) is usually employed to overcome the challenge of sample inefficiency, in which the human expert provides advice for the agent when necessary. The current human-in-the-loop RL (HRL) results mainly focus on discrete action space. In this article, we propose a Q value-dependent policy (QDP)-based HRL (QDP-HRL) algorithm for continuous action space. Considering the cognitive costs of human monitoring, the human expert only selectively gives advice in the early stage of agent learning, where the agent implements human-advised action instead. The QDP framework is adapted to the twin delayed deep deterministic policy gradient algorithm (TD3) in this article for the convenience of comparison with the state-of-the-art TD3. Specifically, the human expert in the QDP-HRL considers giving advice in the case that the difference between the twin Q -networks' output exceeds the maximum difference in the current queue. Moreover, to guide the update of the critic network, the advantage loss function is developed using expert experience and agent policy, which provides the learning direction for the QDP-HRL algorithm to some extent. To verify the effectiveness of QDP-HRL, the experiments are conducted on several continuous action space tasks in the OpenAI gym environment, and the results demonstrate that QDP-HRL greatly improves learning speed and performance.

computer science, artificial intelligence, theory & methods,engineering, electrical & electronic, hardware & architecture
Open-World Reinforcement Learning over Long Short-Term Imagination

Jiajian Li,Qi Wang,Yunbo Wang,Xin Jin,Yang Li,Wenjun Zeng,Xiaokang Yang

2024-10-05

Abstract:Training visual reinforcement learning agents in a high-dimensional open world presents significant challenges. While various model-based methods have improved sample efficiency by learning interactive world models, these agents tend to be "short-sighted", as they are typically trained on short snippets of imagined experiences. We argue that the primary obstacle in open-world decision-making is improving the efficiency of off-policy exploration across an extensive state space. In this paper, we present LS-Imagine, which extends the imagination horizon within a limited number of state transition steps, enabling the agent to explore behaviors that potentially lead to promising long-term feedback. The foundation of our approach is to build a long short-term world model. To achieve this, we simulate goal-conditioned jumpy state transitions and compute corresponding affordance maps by zooming in on specific areas within single images. This facilitates the integration of direct long-term values into behavior learning. Our method demonstrates significant improvements over state-of-the-art techniques in MineDojo.

Machine Learning
Active Exploration Deep Reinforcement Learning for Continuous Action Space with Forward Prediction

Dongfang Zhao,Xu Huanshi,Zhang Xun

DOI: https://doi.org/10.1007/s44196-023-00389-1

IF: 2.259

2024-01-10

International Journal of Computational Intelligence Systems

Abstract:The application of reinforcement learning (RL) to the field of autonomous robotics has high requirements about sample efficiency, since the agent expends for interaction with the environment. One method for sample efficiency is to extract knowledge from existing samples and used to exploration. Typical RL algorithms achieve exploration using task-specific knowledge or adding exploration noise. These methods are limited to current policy improvement level and lack of long-term planning. We propose a novel active exploration deep RL algorithm for the continuous action space problem named active exploration deep reinforcement learning (AEDRL). Our method uses the Gaussian process to model dynamic model, enabling the probability description of prediction sample. Action selection is formulated as the solution of the optimization problem. Thus, the optimization objective is specifically designed for selecting samples that can minimize the uncertainty of the dynamic model. Active exploration is achieved through long-term optimized action selection. This long-term considered action exploration method is more guidance for learning. Enable intelligent agents to explore more interesting action spaces. The proposed AEDRL algorithm is evaluated on several robotic control task including classic pendulum problem and five complex articulated robots. The AEDRL can learn a controller using fewer episodes and demonstrates performance and sample efficiency.

computer science, artificial intelligence, interdisciplinary applications

Investigating Reinforcement Learning Agents for Continuous State Space Environments

Continuous control with deep reinforcement learning

Investigating the Impact of Choice on Deep Reinforcement Learning for Space Controls

Learning Agents With Prioritization and Parameter Noise in Continuous State and Action Space

Addressing High-dimensional Continuous Action Space via Decomposed Discrete Policy-Critic

Reinforcement Learning for Improving Agent Design

Double A3C: Deep Reinforcement Learning on OpenAI Gym Games

Deep Reinforcement Learning with Embedded LQR Controllers

A further exploration of deep Multi-Agent Reinforcement Learning with Hybrid Action Space

DQN with model-based exploration: efficient learning on environments with sparse rewards

Dynamic Path Planning of Unknown Environment Based on Deep Reinforcement Learning

Continuous Control of an Underground Loader Using Deep Reinforcement Learning

Exploring Exploration: Comparing Children with RL Agents in Unified Environments

Continual Reinforcement Learning in 3D Non-stationary Environments

Deep Reinforcement Learning in Parameterized Action Space

Action Mapping for Reinforcement Learning in Continuous Environments with Constraints

Pathfinding in Random Partially Observable Environments with Vision-Informed Deep Reinforcement Learning

Architecting and Visualizing Deep Reinforcement Learning Models

Human-in-the-Loop Reinforcement Learning in Continuous-Action Space

Open-World Reinforcement Learning over Long Short-Term Imagination

Active Exploration Deep Reinforcement Learning for Continuous Action Space with Forward Prediction