Abstract:In large-scale metropolis, it is critical to efficiently allocate various resources such as electricity, medical care, and transportation to meet the living demands of citizens, according to the spatio-temporal distributions of resources and demands. Previous researchers have done plentiful work on such problems by leveraging Multi-Agent Reinforcement Learning (MARL) methods, where multiple agents cooperatively regulate and allocate the resources to meet the demands. However, facing the great number of agents in large cities, existing MARL methods lack efficient parameter sharing strategies among agents to reduce computational complexity. There remain two primary challenges in efficient parameter sharing: (1) during the RL training process, the behavior of agents changes significantly, limiting the performance of group parameter sharing based on fixed role division decided before training; (2) the behavior of agents forms complicated action trajectories, where their role characteristics are implicit, adding difficulty to dynamically adjusting agent role divisions during the training process. In this paper, we propose Dynamic Parameter Sharing (DyPS) to solve the above challenges. We design self-supervised learning tasks to extract the implicit behavioral characteristics from the action trajectories of agents. Based on the obtained behavioral characteristics, we propose a hierarchical MARL framework capable of dynamically revising the agent role divisions during the training process and thus shares parameters among agents with the same role, reducing computational complexity. In addition, our framework can be combined with various typical MARL algorithms, including IPPO, MAPPO, etc. We conduct 7 experiments in 4 representative resource allocation scenarios, where extensive results demonstrate our method's superior performance, outperforming the state-of-the-art baseline methods by up to 31%. Our source codes are available at https://github.com/tsinghua-fib-lab/DyPS.

SEA-PARAM: Exploring Schedulers in Parametric MDPs

Experiences With Scheduling And Mapping Games For Adaptive Distributed Systems: Summary

Optimal Time-Abstract Schedulers for CTMDPs and Markov Games

Pareto Curves for Compositionally Model Checking String Diagrams of MDPs

Online Planning in POMDPs with State-Requests

Parameter Synthesis for Markov Models: Covering the Parameter Space

Hybrid Planning for Dynamic Multimodal Stochastic Shortest Paths

Unpredictable Planning Under Partial Observability

1-2-3-Go! Policy Synthesis for Parameterized Markov Decision Processes via Decision-Tree Learning and Generalization

Search and Explore: Symbiotic Policy Synthesis in POMDPs

Adaptive Online Packing-guided Search for POMDPs

Policy Search for the Optimal Control of Markov Decision Processes: A Novel Particle-Based Iterative Scheme

Sequential Fair Resource Allocation under a Markov Decision Process Framework

Parameterized Markov Decision Process and Its Application to Service Rate Control.

Simulation Optimization Algorithm for SMDPs with Parameterized Randomized Stationary Policies

Sound Heuristic Search Value Iteration for Undiscounted POMDPs with Reachability Objectives

Monte Carlo Planning for Stochastic Control on Constrained Markov Decision Processes

DyPS: Dynamic Parameter Sharing in Multi-Agent Reinforcement Learning for Spatio-Temporal Resource Allocation

Parametric schedulability analysis of a launcher flight control system under reactivity constraints

Capacity-Aware Planning and Scheduling in Budget-Constrained Monotonic MDPs: A Meta-RL Approach