Abstract:This paper presents the hierarchical Q-learning path planning (HQPP) architecture for solving the cooperative tracking control problem of multi-agent systems (MASs) with lumped uncertainties in an unknown environment. The presented architecture consists of three layers, namely, the decision layer, the distributed estimated layer, and the local control layer. Specifically, in the decision layer, we propose the dynamic parameter and trajectory fitting Q-learning (DPTF-Q-learning) algorithm to find a feasible continuous trajectory to the target in an unknown environment. In addition, two dynamic parameters are proposed and introduced into the DPTF-Q-learning algorithm to shorten the required minimum number of steps in the training process. Then, the distributed estimated layer is designed to broadcast the continuous trajectory generated from the decision layer based on the directed communication topology containing a spanning tree. In the local control layer, the cooperative tracking control (CTC) algorithm is proposed to achieve cooperative tracking for MASs in the presence of uncertain dynamics and external disturbances. The sufficient conditions for achieving cooperative tracking control are rigorously derived by employing Lyapunov argument. Finally, numerical simulations are presented to verify the effectiveness of the proposed architecture. Note to Practitioners —This paper is motivated by the need of developing an integrated path planning and control method for cooperative tracking of multi-agent systems in a no-signal environment and without the presence of users. Most related works are limited to separate fields: 1) most existing path planning techniques are only applicable to a single agent and discrete environments, and 2) most existing cooperative tracking algorithms focus on guaranteeing control stability and error convergence without decision-making capabilities. To address the above issues, this work proposes a hierarchical control architecture based on reinforcement learning for multi-agent systems to achieve path planning and cooperative tracking tasks. In addition, multi-agent systems exhibit strong robustness and fault tolerance due to their inherent characteristics, so the above mentioned research can be well applied to post-disaster rescue, intelligent logistics, future war, and so on. Numerical simulations based on Matlab and Python verify the effectiveness of the proposed architecture.

Learning Hierarchical Control For Multi-Agent Capacity-Constrained Systems

Learning-Based Hierarchical Model Predictive Control for Drift Vehicles

Learning Hierarchical Control Systems for Autonomous Systems with Energy Constraints

Model Predictive Control for Multi-Agent Systems under Limited Communication and Time-Varying Network Topology

Learning Hierarchical Behavior and Motion Planning for Autonomous Driving.

Unified Multi-Rate Control: from Low Level Actuation to High Level Planning

Embedded Hierarchical MPC for Autonomous Navigation

Active Learning-based Model Predictive Coverage Control

Moving Forward in Formation: A Decentralized Hierarchical Learning Approach to Multi-Agent Moving Together

Federated Control with Hierarchical Multi-Agent Deep Reinforcement Learning

Hierarchical Multi-Agent Reinforcement Learning for Air Combat Maneuvering

Efficient Motion Control for Heterogeneous Autonomous Vehicle Platoon Using Multilayer Predictive Control Framework

Hierarchical Learning with Heuristic Guidance for Multi-task Assignment and Distributed Planning in Interactive Scenarios

Hierarchical Distributed MPC for Longitudinal and Lateral Vehicle Platoon Control with Collision Avoidance

Hierarchical Control Strategy for Moving A Robot Manipulator Between Small Containers

Hierarchical MPC-based Motion Planning for Automated Vehicles in Parallel Autonomy

Hierarchical Control for Cooperative Teams in Competitive Autonomous Racing

Distributed hierarchical MPC based on evolutionary game for formation control with collision and obstacle avoidance

Hierarchical Incremental MPC for Redundant Robots: A Robust and Singularity-Free Approach

Scalable Multi-modal Model Predictive Control via Duality-based Interaction Predictions

Hierarchical Q-Learning Path Planning for Cooperative Tracking Control of Multi-Agent Systems with Lumped Uncertainties