Abstract:Iterative combinatorial auctions are widely used in high stakes settings such as spectrum auctions. Such auctions can be hard to analyze, making it difficult for bidders to determine how to behave and for designers to optimize auction rules to ensure desirable outcomes such as high revenue or welfare. In this paper, we investigate whether multi-agent reinforcement learning (MARL) algorithms can be used to understand iterative combinatorial auctions, given that these algorithms have recently shown empirical success in several other domains. We find that MARL can indeed benefit auction analysis, but that deploying it effectively is nontrivial. We begin by describing modelling decisions that keep the resulting game tractable without sacrificing important features such as imperfect information or asymmetry between bidders. We also discuss how to navigate pitfalls of various MARL algorithms, how to overcome challenges in verifying convergence, and how to generate and interpret multiple equilibria. We illustrate the promise of our resulting approach by using it to evaluate a specific rule change to a clock auction, finding substantially different auction outcomes due to complex changes in bidders' behavior.

What problem does this paper attempt to address?

### Problems the Paper Attempts to Solve This paper aims to explore whether multi-agent reinforcement learning (MARL) algorithms can be used to understand and analyze Iterative Combinatorial Auctions (ICAs). Specifically, the paper focuses on the following issues: 1. **Auction Complexity**: Iterative combinatorial auctions are very complex in practical applications, especially in high-stakes scenarios like spectrum auctions. The design of these auctions makes bidders' behavior difficult to predict, and designers find it challenging to optimize auction rules to ensure ideal auction outcomes (such as high revenue or high welfare). 2. **Limitations of Existing Methods**: Traditional economic theory methods (such as Bayesian-Nash equilibrium analysis) face significant challenges in dealing with real-world iterative combinatorial auctions because these auctions are too complex to be analyzed with pen and paper. While static simulation methods can observe the impact of rule changes, they usually cannot capture the multiple rational strategies of bidders. 3. **Potential of MARL Applications**: Although MARL has achieved significant success in other fields (such as Go, Shogi, and strategy games), its application in the auction domain still faces many challenges, including how to model the auction process, how to handle multiple equilibrium issues, and how to verify the convergence of the algorithm. ### Main Contributions of the Paper 1. **Methodology**: The paper proposes a general methodology that simplifies auction models (such as abstract action spaces, controlling the number of bidders, and limiting auction length) to enable MARL to be applied to complex iterative combinatorial auctions. The paper also discusses the advantages and disadvantages of different MARL algorithms, as well as how to handle multiple equilibrium issues and verify the convergence of the algorithm. 2. **Case Study**: The paper demonstrates the application of MARL in auction analysis through a specific case study. The study focuses on a key design choice in clock auctions: how to handle bids when multiple bidders want to reduce their demand for the same product. The study finds that MARL can reveal the substantial impact of rule changes on bidder behavior, auction revenue, and auction length. 3. **Tools and Datasets**: The paper contributes a highly configurable clock auction environment that can serve as a test platform for future research. This environment provides many parameters representing possible auction rule changes and generates auction instances and bidder values based on real value models from the literature. ### Summary By introducing multi-agent reinforcement learning technology, the paper provides a new method for understanding and optimizing iterative combinatorial auctions. This method not only helps bidders formulate more effective strategies but also assists designers in evaluating the economic impact of different rule changes. Despite some challenges, the application prospects of MARL in the auction field are broad, and it is expected to provide important support for future auction design and analysis.

Understanding Iterative Combinatorial Auction Designs via Multi-Agent Reinforcement Learning

Deep Reinforcement Learning for Sequential Combinatorial Auctions

Artificial Intelligence for Multi-Unit Auction design

Using Multi-Agent Reinforcement Learning in Auction Simulations

Put Your Money Where Your Mouth Is: Evaluating Strategic Planning and Execution of LLM Agents in an Auction Arena

A Meta-Game Evaluation Framework for Deep Multiagent Reinforcement Learning

Learning to Bid Long-Term: Multi-Agent Reinforcement Learning with Long-Term and Sparse Reward in Repeated Auction Games

Applying Opponent Modeling for Automatic Bidding in Online Repeated Auctions

Learning and Calibrating Heterogeneous Bounded Rational Market Behaviour with Multi-Agent Reinforcement Learning

Formal Contracts Mitigate Social Dilemmas in Multi-Agent RL

A Multi-Agent Reinforcement Learning Method for Impression Allocation in Online Display Advertising

Equilibrium Computation in Multi-Stage Auctions and Contests

Neural Auto-Curricula

Formal contracts mitigate social dilemmas in multi-agent reinforcement learning

Use of Markov Chains to Design an Agent Bidding Strategy for Continuous Double Auctions

Machine Learning-Powered Combinatorial Clock Auction

Using Reinforcement Learning to Validate Empirical Game-Theoretic Analysis: A Continuous Double Auction Study

Combining Tree-Search, Generative Models, and Nash Bargaining Concepts in Game-Theoretic Reinforcement Learning

Equilibrium Learning in Combinatorial Auctions: Computing Approximate Bayesian Nash Equilibria via Pseudogradient Dynamics

Revisiting Some Common Practices in Cooperative Multi-Agent Reinforcement Learning