LongAgent: Scaling Language Models to 128k Context through Multi-Agent Collaboration

Jun Zhao,Can Zu,Hao Xu,Yi Lu,Wei He,Yiwen Ding,Tao Gui,Qi Zhang,Xuanjing Huang

2024-02-18

Abstract:Large language models (LLMs) have demonstrated impressive performance in understanding language and executing complex reasoning tasks. However, LLMs with long context windows have been notorious for their expensive training costs and high inference latency. Even the most advanced models such as GPT-4 and Claude2 often make mistakes when processing inputs of over $100k$ tokens, a phenomenon also known as \textit{lost in the middle}. In this paper, we propose \textsc{LongAgent}, a method based on multi-agent collaboration, which scales LLMs (e.g., LLaMA) to a context of 128K and demonstrates potential superiority in long-text processing compared to GPT-4. In \textsc{LongAgent}, a leader is responsible for understanding user intent and directing team members to acquire information from documents. Due to members' hallucinations, it is non-trivial for a leader to obtain accurate information from the responses of dozens to hundreds of members. To address this, we develop an \textit{inter-member communication} mechanism to resolve response conflicts caused by hallucinations through information sharing. Our experimental results indicate that \textsc{LongAgent} offers a promising alternative for long-text processing. The agent team instantiated with LLaMA-7B achieves significant improvements in tasks such as 128k-long text retrieval, multi-hop question answering, compared to GPT-4.

Artificial Intelligence,Computation and Language

What problem does this paper attempt to address?

The main problem addressed in this paper is the issues that large language models (LLMs) encounter when processing long texts. Due to the quadratic complexity of the attention mechanism, it leads to high training costs and inference delays. When the input text exceeds 100k tokens, the model's performance significantly deteriorates, resulting in the phenomenon of "getting lost in the middle". The paper proposes a method called LONG AGENT, which extends LLMs to a 128K context window through multi-agent collaboration to improve their ability to handle long texts, and compares it with advanced models like GPT-4.

LONG AGENT consists of a leader and multiple members. The leader is responsible for understanding the user's intent and organizing the members to acquire information and resolve conflicts caused by model hallucinations. Information sharing between members is used to address response conflicts. Experimental results show that LONG AGENT outperforms GPT-4 in tasks such as 128k long text retrieval and multi-hop question answering, indicating the potential of multi-agent collaboration in handling long texts. In addition, the paper proposes a larger-scale benchmark test called Needle in a Haystack PLUS to comprehensively evaluate the long text processing capability of LLMs.