Abstract:Modern large language models (LLMs) like ChatGPT have shown remarkable performance on general language tasks but still struggle on complex reasoning tasks, which drives the research on cognitive behaviors of LLMs to explore human-like problem-solving strategies. Along this direction, one representative strategy is self-reflection, which asks an LLM to refine the solution with the feedback generated by itself iteratively. However, our study shows that such reflection-style methods suffer from the Degeneration-of-Thought (DoT) problem: once the LLM has established confidence in its solutions, it is unable to generate novel thoughts later through reflection even if its initial stance is incorrect. To address the DoT problem, we propose a Multi-Agent Debate (MAD) framework, in which multiple agents express their arguments in the state of "tit for tat" and a judge manages the debate process to obtain a final solution. Clearly, our MAD framework encourages divergent thinking in LLMs which would be helpful for tasks that require deep levels of contemplation. Experiment results on two challenging datasets, commonsense machine translation and counter-intuitive arithmetic reasoning, demonstrate the effectiveness of our MAD framework. Extensive analyses suggest that the adaptive break of debate and the modest level of "tit for tat" state are required for MAD to obtain good performance. Moreover, we find that LLMs might not be a fair judge if different LLMs are used for agents. Code is available at <a class="link-external link-https" href="https://github.com/Skytliang/Multi-Agents-Debate" rel="external noopener nofollow">this https URL</a>.

PeerGPT: Probing the Roles of LLM-based Peer Agents as Team Moderators and Participants in Children's Collaborative Learning

AgentPeerTalk: Empowering Students through Agentic-AI-Driven Discernment of Bullying and Joking in Peer Interactions in Schools

When Young Scholars Cooperate with LLMs in Academic Tasks: The Influence of Individual Differences and Task Complexities

Evaluating and Enhancing LLMs Agent based on Theory of Mind in Guandan: A Multi-Player Cooperative Game under Imperfect Information

Prompting and Evaluating Large Language Models for Proactive Dialogues: Clarification, Target-guided, and Non-collaboration

Exploring Collaboration Mechanisms for LLM Agents: A Social Psychology View

MetaAgents: Simulating Interactions of Human Behaviors for LLM-based Task-oriented Coordination via Collaborative Generative Agents

Rethinking Conversational Agents in the Era of LLMs: Proactivity, Non-collaborativity, and Beyond.

Learning Agent-based Modeling with LLM Companions: Experiences of Novices and Experts Using ChatGPT & NetLogo Chat

Evaluating LLMs Capabilities Towards Understanding Social Dynamics

MetaGPT: Meta Programming for A Multi-Agent Collaborative Framework

LLM-Based Agent Society Investigation: Collaboration and Confrontation in Avalon Gameplay

Embodied LLM Agents Learn to Cooperate in Organized Teams

Generating Situated Reflection Triggers about Alternative Solution Paths: A Case Study of Generative AI for Computer-Supported Collaborative Learning

Encouraging Divergent Thinking in Large Language Models through Multi-Agent Debate

Adaptive In-conversation Team Building for Language Model Agents

Predicting challenge moments from students' discourse: A comparison of GPT-4 to two traditional natural language processing approaches

MetaGPT: Meta Programming for Multi-Agent Collaborative Framework

Evil Geniuses: Delving into the Safety of LLM-based Agents

Towards A Human-in-the-Loop LLM Approach to Collaborative Discourse Analysis