Hierarchical Reinforcement Learning Framework towards Multi-agent Navigation

Wenhao Ding,Shuaijun Li,Huihuan Qian

DOI: https://doi.org/10.48550/arXiv.1807.05424

2018-07-17

Abstract:In this paper, we propose a navigation algorithm oriented to multi-agent environment. This algorithm is expressed as a hierarchical framework that contains a Hidden Markov Model (HMM) and a Deep Reinforcement Learning (DRL) structure. For simplification, we term our method Hierarchical Navigation Reinforcement Network (HNRN). In high- level architecture, we train an HMM to evaluate the agent's perception to obtain a score. According to this score, adaptive control action will be chosen. While in low-level architecture, two sub-systems are introduced, one is a differential target- driven system, which aims at heading to the target; the other is a collision avoidance DRL system, which is used for avoiding dynamic obstacles. The advantage of this hierarchical structure is decoupling the target-driven and collision avoidance tasks, leading to a faster and more stable model to be trained. The experiments indicate that our algorithm has higher learning efficiency and rate of success than traditional Velocity Obstacle (VO) algorithms or hybrid DRL method.

Robotics,Multiagent Systems

What problem does this paper attempt to address?

The problem that this paper attempts to solve is to implement an effective navigation algorithm in a multi - agent environment, especially in the dynamic obstacle avoidance task. At present, significant progress has been made in the research on static obstacle scenarios, but there are still great challenges in dynamic obstacle avoidance. In addition, dynamic - obstacle - avoidance navigation has broad application prospects in the interactive scenarios of autonomous vehicles and the collaborative scenarios of logistics warehouses. The paper proposes a new navigation algorithm - the Hierarchical Navigation Reinforcement Network (HNRN), aiming to solve the navigation problems in multi - agent environments. HNRN consists of two levels: 1. **High - level architecture**: Use the Hidden Markov Model (HMM) to evaluate the agent's perception of the environment and obtain a score. Based on this score, select an adaptive control action. 2. **Low - level architecture**: Introduce two subsystems. One is the differential goal - driven system, which is used to move towards the goal; the other is the collision - avoidance system based on Deep Reinforcement Learning (DRL), which is used to avoid obstacles in a multi - agent environment. Through this hierarchical structure, HNRN can decouple the goal - driven and collision - avoidance tasks, thereby speeding up the model training speed and improving stability. The experimental results show that, compared with the traditional Velocity Obstacle (VO) algorithm or the hybrid DRL method, HNRN exhibits higher performance in terms of learning efficiency and success rate.

Hierarchical Reinforcement Learning Framework towards Multi-agent Navigation

Predictive Hierarchical Reinforcement Learning for Path-Efficient Mapless Navigation with Moving Target.

Mapless Collaborative Navigation for a Multi-Robot System Based on the Deep Reinforcement Learning

Hierarchical reinforcement learning for handling sparse rewards in multi-goal navigation

Hierarchical and Stable Multiagent Reinforcement Learning for Cooperative Navigation Control

Learning Hierarchical Behavior and Motion Planning for Autonomous Driving.

Multi-agent navigation based on deep reinforcement learning and traditional pathfinding algorithm

Mobile Robot Navigation Control Based on Hierarchical Reinforcement Learning

Hierarchical Reinforcement Learning with Opponent Modeling for Distributed Multi-agent Cooperation

A Hierarchical Reinforcement Learning Algorithm Based on Attention Mechanism for UAV Autonomous Navigation

Holistic Deep-Reinforcement-Learning-based Training of Autonomous Navigation Systems

Vision-Based Robot Navigation through Combining Unsupervised Learning and Hierarchical Reinforcement Learning

Hierarchical Heterogeneous Multi-Agent Cross-Domain Search Method Based on Deep Reinforcement Learning

Deep reinforcement learning-aided autonomous navigation with landmark generators

HRL4IN: Hierarchical Reinforcement Learning for Interactive Navigation with Mobile Manipulators

Danger-aware Adaptive Composition of DRL Agents for Self-navigation

Efficient Multi-agent Cooperative Navigation in Unknown Environments with Interlaced Deep Reinforcement Learning.

Autonomous Navigation for Cellular-Connected UAV in Highly Dynamic Environments: A Deep Reinforcement Learning Approach

Multi-robot Navigation with Graph Attention Neural Network and Hierarchical Motion Planning

Reinforcement Learned Distributed Multi-Robot Navigation With Reciprocal Velocity Obstacle Shaped Rewards

Hierarchical Reinforcement Learning for Dynamic Autonomous Vehicle Navigation at Intelligent Intersections