ACC-Debate: An Actor-Critic Approach to Multi-Agent Debate

Andrew Estornell,Jean-Francois Ton,Yuanshun Yao,Yang Liu

2024-10-31

Abstract:Large language models (LLMs) have demonstrated a remarkable ability to serve as general-purpose tools for various language-based tasks. Recent works have demonstrated that the efficacy of such models can be improved through iterative dialog between multiple models, frequently referred to as multi-agent debate (MAD). While debate shows promise as a means of improving model efficacy, most works in this area treat debate as an emergent behavior, rather than a learned behavior. In doing so, current debate frameworks rely on collaborative behaviors to have been sufficiently trained into off-the-shelf models. To address this limitation, we propose ACC-Debate, an Actor-Critic based learning framework to produce a two-agent team specialized in debate. We demonstrate that ACC-Debate outperforms SotA debate techniques on a wide array of benchmarks.

Computation and Language,Artificial Intelligence

What problem does this paper attempt to address?

The problem this paper attempts to address is: how to improve the performance of large language models (LLMs) on various tasks through collaborative debate by training multiple LLMs. Specifically, existing Multi-Agent Debate (MAD) methods typically rely on off-the-shelf general-purpose LLMs that have not been specifically trained for collaboration. As a result, the effectiveness of these methods is limited by the models' zero-shot or few-shot capabilities. To overcome this limitation, the authors propose a new framework—Actor-Critic Debate (ACC-Debate), which improves the effectiveness of multi-agent debate by jointly training two agents (an actor model responsible for providing answers and a critic model responsible for feedback).

The main contributions of the paper include:
1. Proposing the first framework for jointly training teams of LLMs for debate (Actor-Critic).
2. Introducing a new data generation scheme—"guided debate trajectories," which can efficiently generate high-quality multi-turn training data.
3. Experimental results show that ACC-Debate significantly outperforms existing state-of-the-art methods on multiple benchmarks.

Through these contributions, the paper aims to enhance the effectiveness of multi-agent debate, enabling LLMs to better collaborate and reason when handling complex tasks.