Abstract:This paper presents the hierarchical Q-learning path planning (HQPP) architecture for solving the cooperative tracking control problem of multi-agent systems (MASs) with lumped uncertainties in an unknown environment. The presented architecture consists of three layers, namely, the decision layer, the distributed estimated layer, and the local control layer. Specifically, in the decision layer, we propose the dynamic parameter and trajectory fitting Q-learning (DPTF-Q-learning) algorithm to find a feasible continuous trajectory to the target in an unknown environment. In addition, two dynamic parameters are proposed and introduced into the DPTF-Q-learning algorithm to shorten the required minimum number of steps in the training process. Then, the distributed estimated layer is designed to broadcast the continuous trajectory generated from the decision layer based on the directed communication topology containing a spanning tree. In the local control layer, the cooperative tracking control (CTC) algorithm is proposed to achieve cooperative tracking for MASs in the presence of uncertain dynamics and external disturbances. The sufficient conditions for achieving cooperative tracking control are rigorously derived by employing Lyapunov argument. Finally, numerical simulations are presented to verify the effectiveness of the proposed architecture. Note to Practitioners —This paper is motivated by the need of developing an integrated path planning and control method for cooperative tracking of multi-agent systems in a no-signal environment and without the presence of users. Most related works are limited to separate fields: 1) most existing path planning techniques are only applicable to a single agent and discrete environments, and 2) most existing cooperative tracking algorithms focus on guaranteeing control stability and error convergence without decision-making capabilities. To address the above issues, this work proposes a hierarchical control architecture based on reinforcement learning for multi-agent systems to achieve path planning and cooperative tracking tasks. In addition, multi-agent systems exhibit strong robustness and fault tolerance due to their inherent characteristics, so the above mentioned research can be well applied to post-disaster rescue, intelligent logistics, future war, and so on. Numerical simulations based on Matlab and Python verify the effectiveness of the proposed architecture.

Behavior Learning Based Distributed Tracking Control for Human-in-the-Loop Multi-Agent Systems

Multi-agent consensus tracking with initial state error by iterative learning control

Distributed Formation Control for a Class of Human-in-the-Loop Multiagent Systems

Model‐free learning‐based distributed cooperative tracking control of human‐in‐the‐loop multi‐agent systems

Adaptive Distributed Tracking Control for Markov Jump Multiagent Systems with a Non-Strict Leader

HiTL-based adaptive fuzzy tracking control of MASs: A distributed fixed-time strategy

Human-in-the-loop Distributed Cooperative Tracking Control with Applications to Autonomous Ground Vehicles: A Data-Driven Mixed Iteration Approach

Reinforcement Learning-Based Unknown Reference Tracking Control of HMASs with Nonidentical Communication Delays

Data-Efficient Off-Policy Learning for Distributed Optimal Tracking Control of HMAS with Unidentified Exosystem Dynamics.

A Hierarchical Distributed Data-Driven Adaptive Learning Control for Nonaffine Nonlinear MASs

Cooperative Tracking Control for Nonlinear MASs under Event-Triggered Communication.

Coordinated Tracking of Linear Multi-Agent Systems with a Dynamic Leader: an Iterative Learning Approach

Learning‐based Distributed Adaptive Control of Heterogeneous Multi‐agent Systems with Unknown Leader Dynamics

Distributed Adaptive Tracking Control of Hidden Leader-Follower Multi-Agent Systems with Unknown Parameters

Optimal Tracking Control of Heterogeneous MASs Using Event-Driven Adaptive Observer and Reinforcement Learning

Distributed output formation tracking control of heterogeneous multi-agent systems using reinforcement learning

Distributed H-infinity Tracking Control for Discrete-Time Multi-Agent Systems with a High-Dimensional Leader

Data-Driven Tracking Control for Multi-Agent Systems with Unknown Dynamics Via Multithreading Iterative Q-Learning

Hierarchical Q-Learning Path Planning for Cooperative Tracking Control of Multi-Agent Systems with Lumped Uncertainties

Fully Distributed Tracking Control for Heterogeneous Multiagent Systems with a Noncooperative Leader over Event-Triggered Communication

Distributed Tracking Control for Linear Multiagent Systems with A Leader of Bounded Unknown Input