Implantable cardioverter defibrillators after acute myocardial infarction

R. Liew

DOI: https://doi.org/10.1136/bmj.c5741

2010-11-19

British Medical Journal

Abstract:Evidence suggests no overall survival benefit if inserted within 40 days

What problem does this paper attempt to address?

Multi-USV Dynamic Navigation and Target Capture: A Guided Multi-Agent Reinforcement Learning Approach

Sulemana Nantogma,Shangyan Zhang,Xuewei Yu,Xuyang An,Yang Xu

DOI: https://doi.org/10.3390/electronics12071523

IF: 2.9

2023-03-24

Electronics

Abstract:Autonomous unmanned systems have become an attractive vehicle for a myriad of military and civilian applications. This can be partly attributed to their ability to bring payloads for utility, sensing, and other uses for various applications autonomously. However, a key challenge in realizing autonomous unmanned systems is the ability to perform complex group missions, which require coordination and collaboration among multiple platforms. This paper presents a cooperative navigating task approach that enables multiple unmanned surface vehicles (multi-USV) to autonomously capture a maneuvering target while avoiding both static and dynamic obstacles. The approach adopts a hybrid multi-agent deep reinforcement learning framework that leverages heuristic mechanisms to guide the group mission learning of the vehicles. Specifically, the proposed framework consists of two stages. In the first stage, navigation subgoal sets are generated based on expert knowledge, and a goal selection heuristic model based on the immune network model is used to select navigation targets during training. Next, the selected goals' executions are learned using actor-critic proximal policy optimization. The simulation results with multi-USV target capture show that the proposed approach is capable of abstracting and guiding the unmanned vehicle group coordination learning and achieving a generally optimized mission execution.

engineering, electrical & electronic,computer science, information systems,physics, applied
Mapless Collaborative Navigation for a Multi-Robot System Based on the Deep Reinforcement Learning

Wenzhou Chen,Shizheng Zhou,Zaisheng Pan,Huixian Zheng,Yong Liu

DOI: https://doi.org/10.3390/app9204198

2019-01-01

Applied Sciences

Abstract:Compared with the single robot system, a multi-robot system has higher efficiency and fault tolerance. The multi-robot system has great potential in some application scenarios, such as the robot search, rescue and escort tasks, and so on. Deep reinforcement learning provides a potential framework for multi-robot formation and collaborative navigation. This paper mainly studies the collaborative formation and navigation of multi-robots by using the deep reinforcement learning algorithm. The proposed method improves the classical Deep Deterministic Policy Gradient (DDPG) to address the single robot mapless navigation task. We also extend the single-robot Deep Deterministic Policy Gradient algorithm to the multi-robot system, and obtain the Parallel Deep Deterministic Policy Gradient (PDDPG). By utilizing the 2D lidar sensor, the group of robots can accomplish the formation construction task and the collaborative formation navigation task. The experiment results in a Gazebo simulation platform illustrates that our method is capable of guiding mobile robots to construct the formation and keep the formation during group navigation, directly through raw lidar data inputs.
Multiple Ships Cooperative Navigation and Collision Avoidance using Multi-agent Reinforcement Learning with Communication

Y. Wang,Y. Zhao

2024-10-12

Abstract:In the real world, unmanned surface vehicles (USV) often need to coordinate with each other to accomplish specific tasks. However, achieving cooperative control in multi-agent systems is challenging due to issues such as non-stationarity and partial observability. Recent advancements in Multi-Agent Reinforcement Learning (MARL) provide new perspectives to address these challenges. Therefore, we propose using the multi-agent deep deterministic policy gradient (MADDPG) algorithm with communication to address multiple ships' cooperation problems under partial observability. We developed two tasks based on OpenAI's gym environment: cooperative navigation and cooperative collision avoidance. In these tasks, ships must not only learn effective control strategies but also establish communication protocols with other agents. We analyze the impact of external noise on communication, the effect of inter-agent communication on performance, and the communication patterns learned by the agents. The results demonstrate that our proposed framework effectively addresses cooperative navigation and collision avoidance among multiple vessels, significantly outperforming traditional single-agent algorithms. Agents establish a consistent communication protocol, enabling them to compensate for missing information through shared observations and achieve better coordination.

Robotics,Systems and Control
Scalable-MADDPG-Based Cooperative Target Invasion for a Multi-USV System

Cheng-Cheng Wang,Yu-Long Wang,Peng Shi,Fei Wang

DOI: https://doi.org/10.1109/tnnls.2023.3309689

IF: 14.255

2023-01-01

IEEE Transactions on Neural Networks and Learning Systems

Abstract:This article concentrates on proposing a scalable deep reinforcement learning (DRL) method for a multiple unmanned surface vehicle (multi-USV) system to operate cooperative target invasion. The multi-USV system, which is made up of multiple invaders, needs to invade target areas in a specified time. A novel scalable reinforcement learning (RL) method called Scalable-MADDPG is proposed for the first time. In this method, the scale of the multi-USV system can be changed at any time without interrupting the training process. Then, to mitigate the policy oscillation after applying Scalable-MADDPG, a bi-directional long-short-term memory (Bi-LSTM) network is constructed. Moreover, an improved ϵ -greedy strategy is proposed to help balance the exploration and exploitation in RL. Furthermore, to enhance the robustness of the optimal policy, Ornstein-Uhlenbeck (OU) noise is added in this improved ϵ -greedy strategy during the training process. Finally, the scalable RL method is used to help the multi-USV system perform cooperative target invasion under complex marine environments. The effectiveness of Scalable-MADDPG is demonstrated through three experiments.

computer science, artificial intelligence, theory & methods,engineering, electrical & electronic, hardware & architecture
Multi-USV System Cooperative Underwater Target Search Based on Reinforcement Learning and Probability Map

Yuan Liu,Yan Peng,Min Wang,Jiajia Xie,Rui Zhou

DOI: https://doi.org/10.1155/2020/7842768

IF: 1.43

2020-05-14

Mathematical Problems in Engineering

Abstract:Unmanned surface vehicle (USV) is a robotic system with autonomous planning, driving, and navigation capabilities. With the continuous development of applications, the missions faced by USV are becoming more and more complex, so it is difficult for a single USV to meet the mission requirements. Compared with a single USV, a multi-USV system has some outstanding advantages such as fewer perceptual constraints, larger operation ranges, and stronger operation capability. In the search mission about multiple stationary underwater targets by a multi-USV system in the environment with obstacles, we propose a novel cooperative search algorithm (CSBDRL) based on reinforcement learning (RL) method and probability map method. CSBDRL is composed of the environmental sense module and policy module, which are organized by the “divide and conquer” policy-based architecture. The environmental sense module focuses on providing environmental sense values by using the probability map method. The policy module focuses on learning the optimal policy by using RL method. In CSBDRL, the mission environment is modeled and the corresponding reward function is designed to effectively explore the environment and learning policies. We test CSBDRL in the simulation environment and compare it with other methods. The results prove that compared with other methods, CSBDRL makes the multi-USV system have a higher search efficiency, which can ensure targets are found more quickly and accurately while ensuring the USV avoids obstacles in time during the mission.

engineering, multidisciplinary,mathematics, interdisciplinary applications
Dynamic Obstacle Avoidance for USVs Using Cross-Domain Deep Reinforcement Learning and Neural Network Model Predictive Controller

Jianwen Li,Jalil Chavez-Galaviz,Kamyar Azizzadenesheli,Nina Mahmoudian

DOI: https://doi.org/10.3390/s23073572

IF: 3.9

2023-03-30

Sensors

Abstract:This work presents a framework that allows Unmanned Surface Vehicles (USVs) to avoid dynamic obstacles through initial training on an Unmanned Ground Vehicle (UGV) and cross-domain retraining on a USV. This is achieved by integrating a Deep Reinforcement Learning (DRL) agent that generates high-level control commands and leveraging a neural network based model predictive controller (NN-MPC) to reach target waypoints and reject disturbances. A Deep Q Network (DQN) utilized in this framework is trained in a ground environment using a Turtlebot robot and retrained in a water environment using the BREAM USV in the Gazebo simulator to avoid dynamic obstacles. The network is then validated in both simulation and real-world tests. The cross-domain learning largely decreases the training time (28%) and increases the obstacle avoidance performance (70 more reward points) compared to pure water domain training. This methodology shows that it is possible to leverage the data-rich and accessible ground environments to train DRL agent in data-poor and difficult-to-access marine environments. This will allow rapid and iterative agent development without further training due to the change in environment or vehicle dynamics.

engineering, electrical & electronic,chemistry, analytical,instruments & instrumentation
Underwater Multi-agent Cooperative Formation Hunting Based on Deep Reinforcement Learning

Xiaobo Shi,Meiqin Liu,Shanling Dong,Ronghao Zheng,Ping Wei

DOI: https://doi.org/10.23919/ccc63176.2024.10662795

2024-01-01

Abstract:In addressing the issue of formation hunting and trajectory planning for multi-autonomous underwater vehicles (AUVs) in complex underwater environments, traditional virtual structure algorithms, and leader-follower models exhibit shortcomings in environmental adaptability and vulnerability to single-point failures. To solve this problem, this article establishes a multi-agent reinforcement learning model with continuous state and action spaces, aiming to optimize the success rate and completion time of the formation hunting task. Furthermore, in establishing the simulation environment for underwater multi-AUVs, a reward function module for the formation hunting task is meticulously designed, taking into account various factors including navigation, formation, efficiency, boundary, and collision avoidance. The efficacy of the proposed methodology was substantiated through a comparative analysis involving the artificial potential field method and the proposed deep reinforcement learning algorithm within the simulation environment. Besides, the efficiency of task execution has improved by approximately 10%, with a success rate approaching 100%.
Unmanned Surface Vehicle Aided Maritime Data Collection Using Deep Reinforcement Learning

Na Su,Jun-Bo Wang,Cheng Zeng,Hua Zhang,Min Lin,Geoffrey Ye Li

DOI: https://doi.org/10.1109/jiot.2022.3168589

IF: 10.6

2022-01-01

IEEE Internet of Things Journal

Abstract:Employing unmanned surface vehicles (USVs) as marine data collectors is promising for large-scale environment sensing in remote ocean monitoring network. In this article, we consider a USV-aided marine data collection network, where a USV collects data from multiple monitoring terminals while avoiding collisions with monitoring terminals and obstacles. Aiming at minimizing energy consumption and data loss, we formulate a trajectory optimization problem with practical constraints, including collision avoidance, steering angle, and velocity limitation. The problem is intractable due to the stochastic arrived data and the random emergence and movement of dynamic obstacles. To efficiently solve it, we transform it as a constrained Markov decision process (MDP) problem and address it using a target-oriented double deep ${Q}$ -learning network (D2QN)-based collision avoidance and trajectory planning algorithm. In the proposed algorithm, the USV acts as an agent to explore and learn its trajectory planning policy by utilizing the causal knowledge. Numerical results demonstrate that the performance of the proposed algorithm is superior in terms of successful probability, energy consumption, and data loss.

computer science, information systems,telecommunications,engineering, electrical & electronic
Maximizing UAV Coverage in Maritime Wireless Networks: A Multiagent Reinforcement Learning Approach

Qianqian Wu,Qiang Liu,Zefan Wu,Jiye Zhang

DOI: https://doi.org/10.3390/fi15110369

2023-11-16

Future Internet

Abstract:In the field of ocean data monitoring, collaborative control and path planning of unmanned aerial vehicles (UAVs) are essential for improving data collection efficiency and quality. In this study, we focus on how to utilize multiple UAVs to efficiently cover the target area in ocean data monitoring tasks. First, we propose a multiagent deep reinforcement learning (DRL)-based path-planning method for multiple UAVs to perform efficient coverage tasks in a target area in the field of ocean data monitoring. Additionally, the traditional Multi-Agent Twin Delayed Deep Deterministic policy gradient (MATD3) algorithm only considers the current state of the agents, leading to poor performance in path planning. To address this issue, we introduce an improved MATD3 algorithm with the integration of a stacked long short-term memory (S-LSTM) network to incorporate the historical interaction information and environmental changes among agents. Finally, the experimental results demonstrate that the proposed MATD3-Stacked_LSTM algorithm can effectively improve the efficiency and practicality of UAV path planning by achieving a high coverage rate of the target area and reducing the redundant coverage rate among UAVs compared with two other advanced DRL algorithms.
Edge-conditioned vector basis functions for the analysis and optimization of rectangular waveguide dual-mode filters

J. Bornemann,U. Rosenberg,S. Amari,R. Vahldieck

DOI: https://doi.org/10.1109/MWSYM.1999.780297

1999-06-13

Abstract:The reliable computer-aided design of narrowband dual-mode filters is usually hampered by extensive CPU-time and memory requirements of commercially available software packages. This paper introduces a new concept within the coupled-integral-equations-technique (CIET) which takes into account all edge conditions simultaneously and, therefore, permits the analysis and optimization of such filter components in a timely fashion. A 12.3 GHz four-pole dual-mode filter in rectangular waveguide is chosen as design example. Comparison with HP's HFSS shows good agreement, whereas the mode-matching technique (MMT) did not converge with up to 600 modes. The CIET routine converges with up to 23 edge-conditioned vector basis functions and up to 1750 modal summation terms. Due to its speed, the new approach can also be used for a Monte-Carlo-based tolerance analysis which shows a manufacturing accuracy of 0.02 mm for this critical dual-mode filter example.
Sim-real joint experimental verification for an unmanned surface vehicle formation strategy based on multi-agent deterministic policy gradient and line of sight guidance

Yan Li,Xiaowen Li,Xiangwei Wei,Hao Wang

DOI: https://doi.org/10.1016/j.oceaneng.2023.113661

IF: 5

2023-01-15

Ocean Engineering

Abstract:The formation of multiple Unmanned Surface Vehicles (USVs) is an effective way to extend the capabilities of a single USV to satisfy relatively complex tasks in practice. In this study, we proposed a formation-strategy-based deep reinforcement learning method called Multi-agent Deterministic Policy Gradient (MADDPG) to realize multi-USV formation. In this work, Line of Sight (LOS) guidance is integrated into the formation strategy under a leader-follower scheme. With the advantage of ignoring the dynamic model of the USV, the proposed formation strategy has strong migration potential to be transferred to other multi-agent systems. To evaluate the performance of the multi-USV formation, we designed two different scenarios in line with the practical tasks carried out with the multi-USV system covering observation aperture enhancement with the desired formation and dynamic non-cooperative target roundup. The performance of the proposed multi-USV formation strategy was demonstrated in both a simulation environment and a real-world environment. Compared with other deep reinforcement learning-inspired and traditional approaches, our proposed strategy based on MADDPG achieved a higher task success rate. It also outperformed the Deep Deterministic Policy Gradient (DDPG) in other metrics because it can acquire knowledge more effectively from dynamic environments by observing joint information and from the centralized training.

engineering, civil, ocean, marine,oceanography
Control and Coordination of a SWARM of Unmanned Surface Vehicles using Deep Reinforcement Learning in ROS

Shrudhi R S,Sreyash Mohanty,Susan Elias

DOI: https://doi.org/10.48550/arXiv.2304.08189

2023-04-17

Abstract:An unmanned surface vehicle (USV) can perform complex missions by continuously observing the state of its surroundings and taking action toward a goal. A SWARM of USVs working together can complete missions faster, and more effectively than a single USV alone. In this paper, we propose an autonomous communication model for a swarm of USVs. The goal of this system is to implement a software system using Robot Operating System (ROS) and Gazebo. With the main objective of coordinated task completion, the Markov decision process (MDP) provides a base to formulate a task decision problem to achieve efficient localization and tracking in a highly dynamic water environment. To coordinate multiple USVs performing real-time target tracking, we propose an enhanced multi-agent reinforcement learning approach. Our proposed scheme uses MA-DDPG, or Multi-Agent Deep Deterministic Policy Gradient, an extension of the Deep Deterministic Policy Gradients (DDPG) algorithm that allows for decentralized control of multiple agents in a cooperative environment. MA-DDPG's decentralised control allows each and every agent to make decisions based on its own observations and objectives, which can lead to superior gross performance and improved stability. Additionally, it provides communication and coordination among agents through the use of collective readings and rewards.

Robotics
Dynamic Target Assignment by Unmanned Surface Vehicles Based on Reinforcement Learning

Tao Hu,Xiaoxue Zhang,Xueshan Luo,Tao Chen

DOI: https://doi.org/10.3390/math12162557

IF: 2.4

2024-08-20

Mathematics

Abstract:Due to the dynamic complexities of the multi-unmanned vessel target assignment problem at sea, especially when addressing moving targets, traditional optimization algorithms often fail to quickly find an adequate solution. To overcome this, we have developed a multi-agent reinforcement learning algorithm. This approach involves defining a state space, employing preferential experience replay, and integrating self-attention mechanisms, which are applied to a novel offshore unmanned vessel model designed for dynamic target allocation. We have conducted a thorough analysis of strike positions and times, establishing robust mathematical models. Additionally, we designed several experiments to test the effectiveness of the algorithm. The proposed algorithm improves the quality of the solution by at least 30% in larger scale scenarios compared to the genetic algorithm (GA), and the average solution speed is less than 10% of the GA, demonstrating the feasibility of the algorithm in solving the problem.

mathematics
Proximal policy optimization with reciprocal velocity obstacle based collision avoidance path planning for multi-unmanned surface vehicles

Delai Xue,Defeng Wu,Andre S. Yamashita,Zhixiong Li

DOI: https://doi.org/10.1016/j.oceaneng.2023.114005

IF: 5

2023-02-26

Ocean Engineering

Abstract:The challenge of solving the collision avoidance path planning problem lies in adaptively selecting the optimal agent velocity in complex scenarios full of reciprocal obstacles. To achieve autonomous collision avoidance for unmanned surface vehicles (USVs), a distributed multi-USVs navigation method based on deep reinforcement learning (DRL) is proposed, which combines the concept of reciprocal velocity obstacle (RVO) with a DRL scheme to solve the collision avoidance path planning problem with limited information. The collision avoidance behavior in USV navigation is based on the International Regulations for Preventing Collisions at Sea, and to obtain an improved proximal policy optimization (PPO) algorithm, the RVO algorithm is used to improve the action space and reward function of the PPO algorithm. Gate Recurrent Unit-based neural network is used to map the states of different number of surrounding obstacles directly into actions. The effectiveness of the method in various situations was verified through simulation experiments. It is also shown that the algorithm can accurately determine collision situations, give reasonable collision avoidance behaviors, and achieve effective collision avoidance in complex environments with dynamic and static obstacles. The research can provide a theoretical basis and methodological reference for USV autonomous navigation.

engineering, civil, ocean, marine,oceanography
Using Deep Reinforcement Learning Methods for Autonomous Vessels in 2D Environments

Mohammad Etemad,Nader Zare,Mahtab Sarvmaili,Amilcar Soares,Bruno Brandoli Machado,Stan Matwin

DOI: https://doi.org/10.48550/arXiv.2003.10249

2020-03-23

Abstract:Unmanned Surface Vehicles technology (USVs) is an exciting topic that essentially deploys an algorithm to safely and efficiently performs a mission. Although reinforcement learning is a well-known approach to modeling such a task, instability and divergence may occur when combining off-policy and function approximation. In this work, we used deep reinforcement learning combining Q-learning with a neural representation to avoid instability. Our methodology uses deep q-learning and combines it with a rolling wave planning approach on agile methodology. Our method contains two critical parts in order to perform missions in an unknown environment. The first is a path planner that is responsible for generating a potential effective path to a destination without considering the details of the root. The latter is a decision-making module that is responsible for short-term decisions on avoiding obstacles during the near future steps of USV exploitation within the context of the value function. Simulations were performed using two algorithms: a basic vanilla vessel navigator (VVN) as a baseline and an improved one for the vessel navigator with a planner and local view (VNPLV). Experimental results show that the proposed method enhanced the performance of VVN by 55.31 on average for long-distance missions. Our model successfully demonstrated obstacle avoidance by means of deep reinforcement learning using planning adaptive paths in unknown environments.

Machine Learning,Artificial Intelligence
Multi-Agent Deep Reinforcement Learning Framework Strategized by Unmanned Aerial Vehicles for Multi-Vessel Full Communication Connection

Jiabao Cao,Jinfeng Dou,Jilong Liu,Xuanning Wei,Zhongwen Guo

DOI: https://doi.org/10.3390/rs15164059

IF: 5

2023-08-17

Remote Sensing

Abstract:In the Internet of Vessels (IoV), it is difficult for any unmanned surface vessel (USV) to work as a coordinator to establish full communication connections (FCCs) among USVs due to the lack of communication connections and the complex natural environment of the sea surface. The existing solutions do not include the employment of some infrastructure to establish USVs' intragroup FCC while relaying data. To address this issue, considering the high-dimension continuous action space and state space of USVs, we propose a multi-agent deep reinforcement learning framework strategized by unmanned aerial vehicles (UAVs). UAVs can evaluate and navigate the multi-USV cooperation and position adjustment to establish a FCC. When ensuring FCCs, we aim to improve the IoV's performance by maximizing the USV's communication range and movement fairness while minimizing their energy consumption, which cannot be explicitly expressed in a closed-form equation. We transform this problem into a partially observable Markov game and design a separate actor–critic structure, in which USVs act as actors and UAVs act as critics to evaluate the actions of USVs and make decisions on their movement. An information transition in UAVs facilitates effective information collection and interaction among USVs. Simulation results demonstrate the superiority of our framework in terms of communication coverage, movement fairness, and average energy consumption, and that it can increase communication efficiency by at least 10% compared to DDPG, with the highest exceeding 120% compared to other baselines.

environmental sciences,imaging science & photographic technology,remote sensing,geosciences, multidisciplinary
Sim-to-Real: Mapless Navigation for USVs Using Deep Reinforcement Learning

Ning Wang,Yabiao Wang,Yuming Zhao,Yong Wang,Zhigang Li

DOI: https://doi.org/10.3390/jmse10070895

IF: 2.744

2022-01-01

Journal of Marine Science and Engineering

Abstract:In recent years, mapless navigation using deep reinforcement learning algorithms has shown significant advantages in improving robot motion planning capabilities. However, the majority of past works have focused on aerial and ground robotics, with very little attention being paid to unmanned surface vehicle (USV) navigation and ultimate deployment on real platforms. In response, this paper proposes a mapless navigation method based on deep reinforcement learning for USVs. Specifically, we carefully design the observation space, action space, reward function, and neural network for a navigation policy that allows the USV to reach the destination collision-free when equipped with only local sensors. Aiming at the sim-to-real transfer and slow convergence of deep reinforcement learning, this paper proposes a dynamics-free training and consistency strategy and designs domain randomization and adaptive curriculum learning. The method was evaluated using a range of tests applied to simulated and physical environments and was proven to work effectively in a real navigation environment.
Obstacle avoidance USV in multi-static obstacle environments based on a deep reinforcement learning approach

Dengyao Jiang,Mingzhe Yuan,Junfeng Xiong,Jinchao Xiao,Yong Duan

DOI: https://doi.org/10.1177/00202940231195937

2023-10-14

Measurement and Control

Abstract:Measurement and Control, Ahead of Print. Unmanned surface vehicles (USVs) are intelligent platforms for unmanned surface navigation based on artificial intelligence, motion control, environmental awareness, and other professional technologies. Obstacle avoidance is an important part of its autonomous navigation. Although the USV works in the water environment (e.g. monitoring and tracking, search and rescue scenarios), the dynamic and complex operating environment makes the traditional methods not suitable for solving the obstacle avoidance problem of the USV. In this paper, to address the issue of poor convergence of the Twin Delayed Deep Deterministic policy gradient (TD3) algorithm of Deep Reinforcement Learning (DRL) in an unstructured environment and wave current interference, random walk policy is proposed to deposit the pre-exploration policy of the algorithm into the experience pool to accelerate the convergence of the algorithm and thus achieve USV obstacle avoidance, which can achieve collision-free navigation from any start point to a given end point in a dynamic and complex environment without offline trajectory and track point generation. We design a pre-exploration policy for the environment and a virtual simulation environment for training and testing the algorithm and give the reward function and training method. The simulation results show that our proposed algorithm is more manageable to converge than the original algorithm and can perform better in complex environments in terms of obstacle avoidance behavior, reflecting the algorithm's feasibility and effectiveness.

automation & control systems,instruments & instrumentation
Collaborative task decision-making of multi-UUV in dynamic environments based on deep reinforcement learning

Haomiao Yu,Sijie Zhang

DOI: https://doi.org/10.1080/17445302.2024.2391788

2024-08-25

Ships and Offshore Structures

Abstract:The problem of collaborative task decision-making for unmanned underwater vehicles (UUVs) in an unknown dynamic ocean environment is investigated. In view of the unknown marine environment, a partially observable Markov decision process is designed to achieve partially observable path planning for multi-UUV. Aiming at the problems of long planning time, large amount of data, and insufficient multi-task decision-making capability in the process of multi-UUV underwater collaborative operation, a multi-UUV collaborative task decision-making model based on the multi-agent twin delayed deep deterministic policy gradient (MATD3) algorithm for the dynamic environment is constructed. The implementation of the centralised training with distributed execution (CT-DE) training framework enables each UUV to possess autonomous decision-making capability and ensures task safety under weak communication conditions. The pre-training phase shows that the MATD3 algorithm outperforms the multi-agent deep deterministic policy gradient (MADDPG) algorithm for training multi-UUV autonomous decision-making scenarios in dynamic environments. The simulation experiment results verify that the autonomous decision-making method based on the MATD3 algorithm can effectively solve the multi-UUV collaborative task decision-making problem in unknown dynamic environments, and the task process satisfies the requirements of real-time, safety and economy.

engineering, marine
A Method of Multi-UAV Cooperative Task Assignment Based on Reinforcement Learning

Xiaohu Zhao,Hanli Jiang,Chenyang An,Ruocheng Wu,Yijun Guo,Daquan Yang

DOI: https://doi.org/10.1155/2022/1147819

2022-08-12

Mobile Information Systems

Abstract:With the increasing complexity of UAV application scenarios, the performance of a single UAV cannot meet the mission requirements. Many complex tasks need the cooperation of multiple UAVs. How to coordinate UAV resources becomes the key to mission completion. In this paper, a task model including multiple UAVs and unknown obstacles is constructed, and the model is transformed into a Markov decision process (MDP). In addition, considering the influence of strategies among UAVs, a multiagent reinforcement learning algorithm based on SAC algorithm and centralized training and decentralized execution framework, MA-SAC (Multi-Agent Soft Actor-Critic), is proposed to solve the MDP. Simulation results show that the algorithm can effectively deal with the task allocation problem of multiple UAVs in this scenario, and its performance is better than other multiagent reinforcement learning algorithms.

computer science, information systems,telecommunications

Implantable cardioverter defibrillators after acute myocardial infarction

Multi-USV Dynamic Navigation and Target Capture: A Guided Multi-Agent Reinforcement Learning Approach

Mapless Collaborative Navigation for a Multi-Robot System Based on the Deep Reinforcement Learning

Multiple Ships Cooperative Navigation and Collision Avoidance using Multi-agent Reinforcement Learning with Communication

Scalable-MADDPG-Based Cooperative Target Invasion for a Multi-USV System

Multi-USV System Cooperative Underwater Target Search Based on Reinforcement Learning and Probability Map

Dynamic Obstacle Avoidance for USVs Using Cross-Domain Deep Reinforcement Learning and Neural Network Model Predictive Controller

Underwater Multi-agent Cooperative Formation Hunting Based on Deep Reinforcement Learning

Unmanned Surface Vehicle Aided Maritime Data Collection Using Deep Reinforcement Learning

Maximizing UAV Coverage in Maritime Wireless Networks: A Multiagent Reinforcement Learning Approach

Edge-conditioned vector basis functions for the analysis and optimization of rectangular waveguide dual-mode filters

Sim-real joint experimental verification for an unmanned surface vehicle formation strategy based on multi-agent deterministic policy gradient and line of sight guidance

Control and Coordination of a SWARM of Unmanned Surface Vehicles using Deep Reinforcement Learning in ROS

Dynamic Target Assignment by Unmanned Surface Vehicles Based on Reinforcement Learning

Proximal policy optimization with reciprocal velocity obstacle based collision avoidance path planning for multi-unmanned surface vehicles

Using Deep Reinforcement Learning Methods for Autonomous Vessels in 2D Environments

Multi-Agent Deep Reinforcement Learning Framework Strategized by Unmanned Aerial Vehicles for Multi-Vessel Full Communication Connection

Sim-to-Real: Mapless Navigation for USVs Using Deep Reinforcement Learning

Obstacle avoidance USV in multi-static obstacle environments based on a deep reinforcement learning approach

Collaborative task decision-making of multi-UUV in dynamic environments based on deep reinforcement learning

A Method of Multi-UAV Cooperative Task Assignment Based on Reinforcement Learning