Abstract:Recent advancements in generative Large Language Models(LLMs) have been remarkable, however, the quality of the text generated by these models often reveals persistent issues. Evaluating the quality of text generated by these models, especially in open-ended text, has consistently presented a significant challenge. Addressing this, recent work has explored the possibility of using LLMs as evaluators. While using a single LLM as an evaluation agent shows potential, it is filled with significant uncertainty and instability. To address these issues, we propose the MATEval: A "Multi-Agent Text Evaluation framework" where all agents are played by LLMs like GPT-4. The MATEval framework emulates human collaborative discussion methods, integrating multiple agents' interactions to evaluate open-ended text. Our framework incorporates self-reflection and Chain-of-Thought (CoT) strategies, along with feedback mechanisms, enhancing the depth and breadth of the evaluation process and guiding discussions towards consensus, while the framework generates comprehensive evaluation reports, including error localization, error types and scoring. Experimental results show that our framework outperforms existing open-ended text evaluation methods and achieves the highest correlation with human evaluation, which confirms the effectiveness and advancement of our framework in addressing the uncertainties and instabilities in evaluating LLMs-generated text. Furthermore, our framework significantly improves the efficiency of text evaluation and model iteration in industrial scenarios.

What problem does this paper attempt to address?

The problem that this paper attempts to solve is the difficult problem of evaluating the quality of text generated by large - language models (LLMs). Specifically, existing evaluation methods face significant challenges when dealing with open - ended texts, such as the limitations of automatic evaluation methods in open - ended text generation, and the uncertainty and instability when using a single LLM as an evaluation agent. To address these challenges, the authors propose a multi - agent discussion framework named MATEval. By simulating the process of human collaborative discussion, combining self - reflection and Chain - of - Thought (CoT) strategies, as well as a feedback mechanism, it improves the reliability and efficiency of evaluating the quality of text generated by LLMs. The main contributions of the MATEval framework include: 1. Proposing a multi - agent evaluation framework, which not only improves the reliability of scoring but also provides accurate diagnostic reports, helping to improve model iteration and audit efficiency in industrial scenarios. 2. Innovatively combining self - reflection and Chain - of - Thought strategies into the multi - agent framework and introducing a feedback mechanism at the end of each round of discussion to resolve differences and promote the achievement of consensus. 3. Verifying the effectiveness of this framework and its high correlation with human evaluation through comprehensive experiments on two English and two Chinese story - text datasets. Overall, this paper aims to provide a more effective and reliable method for evaluating the quality of open - ended texts generated by large - language models by developing the MATEval framework, thereby promoting further development in this field.

MATEval: A Multi-Agent Discussion Framework for Advancing Open-Ended Text Evaluation

ChatEval: Towards Better LLM-based Evaluators through Multi-Agent Debate

Collaborative Evaluation: Exploring the Synergy of Large Language Models and Humans for Open-ended Generation Evaluation

M-MAD: Multidimensional Multi-Agent Debate Framework for Fine-grained Machine Translation Evaluation

SAGEval: The frontiers of Satisfactory Agent based NLG Evaluation for reference-free open-ended text

Can Large Language Models be Trusted for Evaluation? Scalable Meta-Evaluation of LLMs as Evaluators via Agent Debate

TencentLLMEval: A Hierarchical Evaluation of Real-World Capabilities for Human-Aligned LLMs

Towards Better Open-Ended Text Generation: A Multicriteria Evaluation Framework

Revisiting Benchmark and Assessment: An Agent-based Exploratory Dynamic Evaluation Framework for LLMs

Towards a Unified Multi-Dimensional Evaluator for Text Generation.

Automatic Evaluation for Text-to-image Generation: Task-decomposed Framework, Distilled Training, and Meta-evaluation Benchmark

MME-Survey: A Comprehensive Survey on Evaluation of Multimodal LLMs

RepEval: Effective Text Evaluation with LLM Representation

A Comprehensive Analysis of the Effectiveness of Large Language Models As Automatic Dialogue Evaluators

CMAT: A Multi-Agent Collaboration Tuning Framework for Enhancing Small Language Models

Dynamic Evaluation of Large Language Models by Meta Probing Agents

AgentBoard: An Analytical Evaluation Board of Multi-turn LLM Agents

DEBATE: Devil's Advocate-Based Assessment and Text Evaluation

Encouraging Divergent Thinking in Large Language Models through Multi-Agent Debate

MT-Eval: A Multi-Turn Capabilities Evaluation Benchmark for Large Language Models