Abstract:Reinforcement learning algorithms require a large amount of samples; this often limits their real-world applications on even simple tasks. Such a challenge is more outstanding in multi-agent tasks, as each step of operation is more costly, requiring communications or shifting or resources. This work aims to improve data efficiency of multi-agent control by model-based learning. We consider networked systems where agents are cooperative and communicate only locally with their neighbors, and propose the decentralized model-based policy optimization framework (DMPO). In our method, each agent learns a dynamic model to predict future states and broadcast their predictions by communication, and then the policies are trained under the model rollouts. To alleviate the bias of model-generated data, we restrain the model usage for generating myopic rollouts, thus reducing the compounding error of model generation. To pertain the independence of policy update, we introduce extended value function and theoretically prove that the resulting policy gradient is a close approximation to true policy gradients. We evaluate our algorithm on several benchmarks for intelligent transportation systems, which are connected autonomous vehicle control tasks (Flow and CACC) and adaptive traffic signal control (ATSC). Empirical results show that our method achieves superior data efficiency and matches the performance of model-free methods using true models. The source code of our algorithm and baselines can be found at https://github.com/PKU-MARL/Model-Based-MARL.

Centralized Optimization for Dec-POMDPs under the Expected Average Reward Criterion

Decentralized Policy Optimization

Finding Optimal Observation-Based Policies for Constrained POMDPs under the Expected Average Reward Criterion

Finding Optimal Memoryless Policies of POMDPs under the Expected Average Reward Criterion

Model-Based Decentralized Policy Optimization

Monte-Carlo Search for an Equilibrium in Dec-POMDPs

Learning for Decentralized Control of Multiagent Systems in Large, Partially-Observable Stochastic Environments

Optimally Solving Simultaneous-Move Dec-POMDPs: The Sequential Central Planning Approach

Privacy-Preserving Push-Pull Method for Decentralized Optimization via State Decomposition

Observation-Based Optimization for POMDPs with Continuous State, Observation, and Action Spaces.

Decentralized Natural Policy Gradient with Variance Reduction for Collaborative Multi-Agent Reinforcement Learning

A Proximal Gradient Algorithm for Decentralized Nondifferentiable Optimization.

A Role-Based POMDPs Approach for Decentralized Implicit Cooperation of Multiple Agents.

Decentralized Policy Gradient for Nash Equilibria Learning of General-sum Stochastic Games

Scalable Model-based Policy Optimization for Decentralized Networked Systems

Gradient-Type Methods For Decentralized Optimization Problems With Polyak-Łojasiewicz Condition Over Time-Varying Networks

Landscape of Policy Optimization for Finite Horizon MDPs with General State and Action

Decentralized Stochastic Optimization Algorithms Using Uncoordinated Step-Sizes over Unbalanced Directed Networks

ADMM Based Privacy-preserving Decentralized Optimization

DIAMOND: Taming Sample and Communication Complexities in Decentralized Bilevel Optimization