Abstract:This paper presents the hierarchical Q-learning path planning (HQPP) architecture for solving the cooperative tracking control problem of multi-agent systems (MASs) with lumped uncertainties in an unknown environment. The presented architecture consists of three layers, namely, the decision layer, the distributed estimated layer, and the local control layer. Specifically, in the decision layer, we propose the dynamic parameter and trajectory fitting Q-learning (DPTF-Q-learning) algorithm to find a feasible continuous trajectory to the target in an unknown environment. In addition, two dynamic parameters are proposed and introduced into the DPTF-Q-learning algorithm to shorten the required minimum number of steps in the training process. Then, the distributed estimated layer is designed to broadcast the continuous trajectory generated from the decision layer based on the directed communication topology containing a spanning tree. In the local control layer, the cooperative tracking control (CTC) algorithm is proposed to achieve cooperative tracking for MASs in the presence of uncertain dynamics and external disturbances. The sufficient conditions for achieving cooperative tracking control are rigorously derived by employing Lyapunov argument. Finally, numerical simulations are presented to verify the effectiveness of the proposed architecture. Note to Practitioners —This paper is motivated by the need of developing an integrated path planning and control method for cooperative tracking of multi-agent systems in a no-signal environment and without the presence of users. Most related works are limited to separate fields: 1) most existing path planning techniques are only applicable to a single agent and discrete environments, and 2) most existing cooperative tracking algorithms focus on guaranteeing control stability and error convergence without decision-making capabilities. To address the above issues, this work proposes a hierarchical control architecture based on reinforcement learning for multi-agent systems to achieve path planning and cooperative tracking tasks. In addition, multi-agent systems exhibit strong robustness and fault tolerance due to their inherent characteristics, so the above mentioned research can be well applied to post-disaster rescue, intelligent logistics, future war, and so on. Numerical simulations based on Matlab and Python verify the effectiveness of the proposed architecture.

Extending Htn To Planning And Execution Control For Small Combat Unit Simulation

Application of Hierarchical Task Network Planning to Uninhabited Combat Aerial Vehicles Planning

Adaptive CGF Commander Behavior Modeling Through HTN Guided Monte Carlo Tree Search

Hierarchical Decision and Control for Continuous Multitarget Problem: Policy Evaluation with Action Delay

Modeling and Simulation of Command and Control System Based on HLA Technology in the Cooperative Air Combat

Hierarchical Control Design for the Cooperative Target Enclosing Motion of Unmanned Surface Vehicle.

Design of Mixed Ground-to-Air Missile Combat Unit Command and Control System Simulation Test-bed

Hierarchical Planning for Agent-Based Traffic Management and Control

HATP: An HTN Planner for Robotics

Hierarchical planning-based crowd formation: HPCF

An Operation Planning Generation and Optimization Method for the New Intelligent Combat SoS

Hierarchical Task Planning for Multiarm Robot with Multiconstraint

A Hybrid Approach to Planning and Execution in Dynamic Environments Through Hierarchical Task Networks and Behavior Trees

Hierarchical Learning with Heuristic Guidance for Multi-task Assignment and Distributed Planning in Interactive Scenarios

Real-World Deployment of a Hierarchical Uncertainty-Aware Collaborative Multiagent Planning System

Hierarchical Task Network-Based Emergency Task Planning with Incomplete Information, Concurrency and Uncertain Duration

Function Chain-Based Mission Planning Method for Hybrid Combat SoS.

Hierarchical Extraction, Planning and Behavior Tree Process Control Based on Historical Trajectory

Mission Decomposition Tactics with Variable Granularity Based on Extended Hierarchical Task Network Planning

Crowd Formation Via Hierarchical Planning

Hierarchical Q-Learning Path Planning for Cooperative Tracking Control of Multi-Agent Systems with Lumped Uncertainties