Abstract:One of the main challenges in offline Reinforcement Learning (RL) is the distribution shift that arises from the learned policy deviating from the data collection policy. This is often addressed by avoiding out-of-distribution (OOD) actions during policy improvement as their presence can lead to substantial performance degradation. This challenge is amplified in the offline Multi-Agent RL (MARL) setting since the joint action space grows exponentially with the number of agents. To avoid this curse of dimensionality, existing MARL methods adopt either value decomposition methods or fully decentralized training of individual agents. However, even when combined with standard conservatism principles, these methods can still result in the selection of OOD joint actions in offline MARL. To this end, we introduce AlberDICE, an offline MARL algorithm that alternatively performs centralized training of individual agents based on stationary distribution optimization. AlberDICE circumvents the exponential complexity of MARL by computing the best response of one agent at a time while effectively avoiding OOD joint action selection. Theoretically, we show that the alternating optimization procedure converges to Nash policies. In the experiments, we demonstrate that AlberDICE significantly outperforms baseline algorithms on a standard suite of MARL benchmarks.

What problem does this paper attempt to address?

This paper attempts to solve the out - of - distribution (OOD) joint action selection problem caused by the curse of dimensionality of the joint action space in offline Multi - Agent Reinforcement Learning (MARL). Specifically: 1. **Challenges in Offline MARL**: - In offline MARL, since the dataset is generated by different data collection strategies, the learned policy may deviate from these data collection strategies, resulting in distribution shift. This shift is especially more severe in multi - agent environments because as the number of agents increases, the joint action space grows exponentially. - Existing methods such as value decomposition or fully decentralized training can avoid the curse of dimensionality to a certain extent, but may still lead to the selection of OOD joint actions, even when combined with the principle of conservatism. 2. **Proposal of AlberDICE**: - To solve the above problems, the authors propose AlberDICE (ALternate BEstResponse Stationary DIstribution Correction Estimation), a new offline MARL algorithm. This algorithm avoids OOD joint action selection by alternately optimizing the strategy of each agent and effectively bypasses the curse of dimensionality of the joint action space. - The core idea of AlberDICE is to calculate the best - response strategy of each agent through centralized training while fixing the strategies of other agents. This method not only considers the non - stationarity brought by the changes of other agents' strategies but also ensures the conservatism of the data distribution through a regularization term. 3. **Theoretical and Experimental Verification**: - Theoretically, the authors prove that the alternating optimization process converges to Nash policies and that the proposed regularization term can maintain the common reward structure of the task. - Experimental results show that AlberDICE significantly outperforms the baseline algorithms in multiple standard MARL benchmark tests, especially in tasks that require a high degree of coordination. In summary, this paper aims to solve the OOD joint action selection problem caused by the curse of dimensionality of the joint action space in offline MARL. This goal is achieved by proposing the AlberDICE algorithm and is verified theoretically and experimentally.

AlberDICE: Addressing Out-Of-Distribution Joint Actions in Offline Multi-Agent RL via Alternating Stationary Distribution Correction Estimation

ComaDICE: Offline Cooperative Multi-Agent Reinforcement Learning with Stationary Distribution Shift Regularization

Bridging Distributionally Robust Learning and Offline RL: An Approach to Mitigate Distribution Shift and Partial Data Coverage

Multi-Agent Reinforcement Learning With Decentralized Distribution Correction

Offline Multi-Agent Reinforcement Learning via In-Sample Sequential Policy Optimization

Out-of-Distribution Adaptation in Offline RL: Counterfactual Reasoning via Causal Normalizing Flows

Coordination Failure in Cooperative Offline MARL

ODICE: Revealing the Mystery of Distribution Correction Estimation via Orthogonal-gradient Update

Offline Multi-Agent Reinforcement Learning with Implicit Global-to-Local Value Regularization

SimuDICE: Offline Policy Optimization Through World Model Updates and DICE Estimation

Scaling Pareto-Efficient Decision Making Via Offline Multi-Objective RL

Adaptable Conservative Q-Learning for Offline Reinforcement Learning.

Counterfactual Conservative Q Learning for Offline Multi-agent Reinforcement Learning

CDSA: Conservative Denoising Score-based Algorithm for Offline Reinforcement Learning

Robust Offline Reinforcement Learning from Low-Quality Data

Offline Decentralized Multi-Agent Reinforcement Learning

UAC: Offline Reinforcement Learning with Uncertain Action Constraint

Combined Constraint on Behavior Cloning and Discriminator in Offline Reinforcement Learning

Decentralized Multi-Agent Reinforcement Learning: An Off-Policy Method

Goal-conditioned Offline Reinforcement Learning through State Space Partitioning

Diffusion Policies for Out-of-Distribution Generalization in Offline Reinforcement Learning