Abstract:Multi-agent systems (MAS) consist of multiple autonomous agents interacting to achieve collective objectives. Multi-agent reinforcement learning (MARL) enhances these systems by enabling agents to learn optimal behaviors through interaction, thus improving their coordination in dynamic environments. However, MARL faces significant challenges in adapting to complex dependencies on past states and actions, which are not adequately represented by the current state alone in reactive systems. This paper addresses these challenges by considering MAS operating under task specifications formulated as Generalized Reactivity of rank 1 (GR(1)). These synthesized strategies are used as a priori knowledge to guide the learning. To tackle the difficulties of handling non-Markovian tasks in reactive systems, we propose a novel synchronized decentralized training paradigm that guides agents to learn within the MARL framework using a reward structure constructed from decomposed synthesized strategies of GR(1). We initially formalize the synthesis of GR(1) strategies as a reachability problem of winning states of the system. Subsequently, we develop a decomposition mechanism that constructs individual reward structures for decentralized MARL, incorporating potential values calculated through value iteration. Theoretical proofs are provided to verify that the safety and liveness properties are preserved. We evaluate our approach against other state-of-the-art methods under various GR(1) specifications and scenario maps, demonstrating superior learning efficacy and optimal rewards per episode. Additionally, we show that the decentralized training paradigm outperforms the centralized training paradigm. The value iteration strategy used to calculate potential values for the reward structure is compared against two other strategies, showcasing its advantages.

Adaptive Learning Rates for Multi-Agent Reinforcement Learning

S2rl

Adaptive Individual Q-Learning-A Multiagent Reinforcement Learning Method for Coordination Optimization

Optimal Exploration Algorithm of Multi-Agent Reinforcement Learning Methods (Student Abstract)

ACE: Cooperative Multi-agent Q-learning with Bidirectional Action-Dependency

Cooperative Multi-Agent Reinforcement Learning with Partial Observations

PPS-QMIX: Periodically Parameter Sharing for Accelerating Convergence of Multi-Agent Reinforcement Learning

MANSA: Learning Fast and Slow in Multi-Agent Systems

Towards Efficient Multi-Agent Learning Systems

Higher Replay Ratio Empowers Sample-Efficient Multi-Agent Reinforcement Learning

Sample-Efficient Multi-Agent RL: an Optimization Perspective.

Multi-agent reinforcement learning with synchronized and decomposed reward automaton synthesized from reactive temporal logic

Efficient Multi-agent Reinforcement Learning by Planning

Decentralized Multi-agent Reinforcement Learning with Multi-time Scale of Decision Epochs

Towards Global Optimality in Cooperative MARL with the Transformation And Distillation Framework

Decentralized Multi-Agent Reinforcement Learning: An Off-Policy Method

Multi-Agent Reinforcement Learning via Adaptive Kalman Temporal Difference and Successor Representation

Multi-Agent Advisor Q-Learning

Multiexperience-Assisted Efficient Multiagent Reinforcement Learning

F2A2: Flexible Fully-decentralized Approximate Actor-critic for Cooperative Multi-agent Reinforcement Learning

Learning Automata-Based Multiagent Reinforcement Learning for Optimization of Cooperative Tasks