Abstract:Most multi-agent reinforcement learning approaches adopt two types of policy optimization methods that either update policy simultaneously or sequentially. Simultaneously updating policies of all agents introduces non-stationarity problem. Although sequentially updating policies agent-by-agent in an appropriate order improves policy performance, it is prone to low efficiency due to sequential execution, resulting in longer model training and execution time. Intuitively, partitioning policies of all agents according to their interdependence and updating joint policy batch-by-batch can effectively balance performance and efficiency. However, how to determine the optimal batch partition of policies and batch updating order are challenging problems. Firstly, a sequential batched policy updating scheme, B2MAPO (Batch by Batch Multi-Agent Policy Optimization), is proposed with a theoretical guarantee of the monotonic incrementally tightened bound. Secondly, a universal modulized plug-and-play B2MAPO hierarchical framework, which satisfies CTDE principle, is designed to conveniently integrate any MARL models to fully exploit and merge their merits, including policy optimality and inference efficiency. Next, a DAG-based B2MAPO algorithm is devised, which is a carefully designed implementation of B2MAPO framework. Comprehensive experimental results conducted on StarCraftII Multi-agent Challenge and Google Football Research demonstrate the performance of DAG-based B2MAPO algorithm outperforms baseline methods. Meanwhile, compared with A2PO, our algorithm reduces the model training and execution time by 60.4% and 78.7%, respectively.

FP3O: Enabling Proximal Policy Optimization in Multi-Agent Cooperation with Parameter-Sharing Versatility

Coordinated Proximal Policy Optimization

The Surprising Effectiveness of PPO in Cooperative Multi-Agent Games

The Surprising Effectiveness of PPO in Cooperative, Multi-Agent Games

JointPPO: Diving Deeper into the Effectiveness of PPO in Multi-Agent Reinforcement Learning

PIPO: Policy Optimization with Permutation-Invariant Constraint for Distributed Multi-Robot Navigation

Policy Regularization via Noisy Advantage Values for Cooperative Multi-agent Actor-Critic methods

Multi-Path Policy Optimization

Proximal Policy Optimization Based Decentralized Networked Multi-Agent Reinforcement Learning

Meta Proximal Policy Optimization for Cooperative Multi-Agent Continuous Control

An Improved PPO for Multiple Unmanned Aerial Vehicles

Beyond the Boundaries of Proximal Policy Optimization

Towards Global Optimality in Cooperative MARL with Sequential Transformation

Authentic Boundary Proximal Policy Optimization

Truly Proximal Policy Optimization

Proximal Policy Optimization with Policy Feedback

Exploration in policy optimization through multiple paths

Communication-Efficient Cooperative Multi-Agent PPO via Regulated Segment Mixture in Internet of Vehicles

Assigning Credit with Partial Reward Decoupling in Multi-Agent Proximal Policy Optimization

B2MAPO: A Batch-by-Batch Multi-Agent Policy Optimization to Balance Performance and Efficiency

Decentralized Policy Optimization