Abstract:Multi-agent systems (MAS) consist of multiple autonomous agents interacting to achieve collective objectives. Multi-agent reinforcement learning (MARL) enhances these systems by enabling agents to learn optimal behaviors through interaction, thus improving their coordination in dynamic environments. However, MARL faces significant challenges in adapting to complex dependencies on past states and actions, which are not adequately represented by the current state alone in reactive systems. This paper addresses these challenges by considering MAS operating under task specifications formulated as Generalized Reactivity of rank 1 (GR(1)). These synthesized strategies are used as a priori knowledge to guide the learning. To tackle the difficulties of handling non-Markovian tasks in reactive systems, we propose a novel synchronized decentralized training paradigm that guides agents to learn within the MARL framework using a reward structure constructed from decomposed synthesized strategies of GR(1). We initially formalize the synthesis of GR(1) strategies as a reachability problem of winning states of the system. Subsequently, we develop a decomposition mechanism that constructs individual reward structures for decentralized MARL, incorporating potential values calculated through value iteration. Theoretical proofs are provided to verify that the safety and liveness properties are preserved. We evaluate our approach against other state-of-the-art methods under various GR(1) specifications and scenario maps, demonstrating superior learning efficacy and optimal rewards per episode. Additionally, we show that the decentralized training paradigm outperforms the centralized training paradigm. The value iteration strategy used to calculate potential values for the reward structure is compared against two other strategies, showcasing its advantages.

Multi-Agent Reinforcement Learning with Reward Delays

Multiagent Reinforcement Learning for Strictly Constrained Tasks Based on Reward Recorder

S2rl

Multi-Agent Reinforcement Learning in Time-varying Networked Systems

Hierarchical Deep Multiagent Reinforcement Learning with Temporal Abstraction

Decentralized Multi-agent Reinforcement Learning with Multi-time Scale of Decision Epochs

Off-Beat Multi-Agent Reinforcement Learning

Fully decentralized multi-agent reinforcement learning with networked agents

Multi-Agent Reinforcement Learning in Stochastic Networked Systems

Multi-Agent Reinforcement Learning in Stochastic Networked Systems.

Multi-agent reinforcement learning with synchronized and decomposed reward automaton synthesized from reactive temporal logic

A finite time analysis of distributed Q-learning

Agent-Time Attention for Sparse Rewards Multi-Agent Reinforcement Learning

A Cooperative Multi-Agent Reinforcement Learning Method Based on Coordination Degree

General multi-agent reinforcement learning integrating heuristic-based delay priority strategy for demand and capacity balancing

Communication-Efficient and Resilient Distributed Q-Learning.

Inducing Cooperation via Team Regret Minimization based Multi-Agent Deep Reinforcement Learning

Adaptive Individual Q-Learning-A Multiagent Reinforcement Learning Method for Coordination Optimization

Towards Efficient Multi-Agent Learning Systems

Scalable Multi-Agent Reinforcement Learning for Networked Systems with Average Reward

Dealing With Non-stationarity in Decentralized Cooperative Multi-Agent Deep Reinforcement Learning via Multi-Timescale Learning