Abstract:In an era where single large language models have dominated the landscape of artificial intelligence for years, multi-agent systems arise as new protagonists in conversational task-solving. While previous studies have showcased their potential in reasoning tasks and creative endeavors, an analysis of their limitations concerning the conversational paradigms and the impact of individual agents is missing. It remains unascertained how multi-agent discussions perform across tasks of varying complexity and how the structure of these conversations influences the process. To fill that gap, this work systematically evaluates multi-agent systems across various discussion paradigms, assessing their strengths and weaknesses in both generative tasks and question-answering tasks. Alongside the experiments, I propose a taxonomy of 20 multi-agent research studies from 2022 to 2024, followed by the introduction of a framework for deploying multi-agent LLMs in conversational task-solving. I demonstrate that while multi-agent systems excel in complex reasoning tasks, outperforming a single model by leveraging expert personas, they fail on basic tasks. Concretely, I identify three challenges that arise: 1) While longer discussions enhance reasoning, agents fail to maintain conformity to strict task requirements, which leads to problem drift, making shorter conversations more effective for basic tasks. 2) Prolonged discussions risk alignment collapse, raising new safety concerns for these systems. 3) I showcase discussion monopolization through long generations, posing the problem of fairness in decision-making for tasks like summarization. This work uncovers both the potential and challenges that arise with multi-agent interaction and varying conversational paradigms, providing insights into how future research could improve the efficiency, performance, and safety of multi-agent LLMs.

Exploring the Intersection of Large Language Models and Agent-Based Modeling via Prompt Engineering

Synergistic Simulations: Multi-Agent Problem Solving with Large Language Models

Smart Agent-Based Modeling: On the Use of Large Language Models in Computer Simulations

Large Language Models Empowered Agent-based Modeling and Simulation: A Survey and Perspectives

Shall We Team Up: Exploring Spontaneous Cooperation of Competing LLM Agents

User Behavior Simulation with Large Language Model based Agents

LMAgent: A Large-scale Multimodal Agents Society for Multi-user Simulation

Sense and Sensitivity: Evaluating the simulation of social dynamics via Large Language Models

Computational Experiments Meet Large Language Model Based Agents: A Survey and Perspective

AMONGAGENTS: Evaluating Large Language Models in the Interactive Text-Based Social Deduction Game

Simulating Human Strategic Behavior: Comparing Single and Multi-agent LLMs

LLM-Augmented Agent-Based Modelling for Social Simulations: Challenges and Opportunities

Simulating Human Society with Large Language Model Agents: City, Social Media, and Economic System

When Large Language Model based Agent Meets User Behavior Analysis: A Novel User Simulation Paradigm

MetaAgents: Simulating Interactions of Human Behaviors for LLM-based Task-oriented Coordination via Collaborative Generative Agents

Reproducing and Extending Experiments in Behavioral Strategy with Large Language Models

S3: Social-network Simulation System with Large Language Model-Empowered Agents

Multi-Agent Large Language Models for Conversational Task-Solving

GenSim: A General Social Simulation Platform with Large Language Model based Agents

LLM experiments with simulation: Large Language Model Multi-Agent System for Simulation Model Parametrization in Digital Twins