Abstract:Recent advancements in generative Large Language Models(LLMs) have been remarkable, however, the quality of the text generated by these models often reveals persistent issues. Evaluating the quality of text generated by these models, especially in open-ended text, has consistently presented a significant challenge. Addressing this, recent work has explored the possibility of using LLMs as evaluators. While using a single LLM as an evaluation agent shows potential, it is filled with significant uncertainty and instability. To address these issues, we propose the MATEval: A "Multi-Agent Text Evaluation framework" where all agents are played by LLMs like GPT-4. The MATEval framework emulates human collaborative discussion methods, integrating multiple agents' interactions to evaluate open-ended text. Our framework incorporates self-reflection and Chain-of-Thought (CoT) strategies, along with feedback mechanisms, enhancing the depth and breadth of the evaluation process and guiding discussions towards consensus, while the framework generates comprehensive evaluation reports, including error localization, error types and scoring. Experimental results show that our framework outperforms existing open-ended text evaluation methods and achieves the highest correlation with human evaluation, which confirms the effectiveness and advancement of our framework in addressing the uncertainties and instabilities in evaluating LLMs-generated text. Furthermore, our framework significantly improves the efficiency of text evaluation and model iteration in industrial scenarios.

Adversarial Evaluation for Models of Natural Language

Training NLI Models Through Universal Adversarial Attack

Unveiling the Achilles' Heel of NLG Evaluators: A Unified Adversarial Framework Driven by Large Language Models

From Adversarial Arms Race to Model-centric Evaluation: Motivating a Unified Automatic Robustness Evaluation Framework

Adversarial Multi-Agent Evaluation of Large Language Models through Iterative Debates

Adversarial Constraint Evaluation on Biomedical Text Mining

Lessons from the Trenches on Reproducible Evaluation of Language Models

MENLI: Robust Evaluation Metrics from Natural Language Inference

Language Model Evaluation Beyond Perplexity

Establishing Trustworthiness: Rethinking Tasks and Model Evaluation

Bring Your Own Data! Self-Supervised Evaluation for Large Language Models

Adversarial Attack and Defense of Structured Prediction Models

Adversarial NLI: A new benchmark for natural language understanding

Rethinking the Evaluating Framework for Natural Language Understanding in AI Systems: Language Acquisition as a Core for Future Metrics

Large Language Models Are Active Critics in NLG Evaluation

MATEval: A Multi-Agent Discussion Framework for Advancing Open-Ended Text Evaluation

Adversarial Attacks and Defenses in Large Language Models: Old and New Threats

Self-critiquing models for assisting human evaluators

On Adversarial Examples for Biomedical NLP Tasks

A Pragmatics-Centered Evaluation Framework for Natural Language Understanding