Abstract:Recent advancements in generative Large Language Models(LLMs) have been remarkable, however, the quality of the text generated by these models often reveals persistent issues. Evaluating the quality of text generated by these models, especially in open-ended text, has consistently presented a significant challenge. Addressing this, recent work has explored the possibility of using LLMs as evaluators. While using a single LLM as an evaluation agent shows potential, it is filled with significant uncertainty and instability. To address these issues, we propose the MATEval: A "Multi-Agent Text Evaluation framework" where all agents are played by LLMs like GPT-4. The MATEval framework emulates human collaborative discussion methods, integrating multiple agents' interactions to evaluate open-ended text. Our framework incorporates self-reflection and Chain-of-Thought (CoT) strategies, along with feedback mechanisms, enhancing the depth and breadth of the evaluation process and guiding discussions towards consensus, while the framework generates comprehensive evaluation reports, including error localization, error types and scoring. Experimental results show that our framework outperforms existing open-ended text evaluation methods and achieves the highest correlation with human evaluation, which confirms the effectiveness and advancement of our framework in addressing the uncertainties and instabilities in evaluating LLMs-generated text. Furthermore, our framework significantly improves the efficiency of text evaluation and model iteration in industrial scenarios.

Towards Effective Automatic Evaluation of Generated Reflections for Motivational Interviewing

Generation, Distillation and Evaluation of Motivational Interviewing-Style Reflections with a Foundational Language Model

Self-Contrast: Better Reflection Through Inconsistent Solving Perspectives

ChatEval: Towards Better LLM-based Evaluators through Multi-Agent Debate

When Hindsight is Not 20/20: Testing Limits on Reflective Thinking in Large Language Models

VERVE: Template-based ReflectiVE Rewriting for MotiVational IntErviewing

Towards Automatic Evaluation of Dialog Systems: A Model-Free Off-Policy Evaluation Approach

An Automatic and Cost-Efficient Peer-Review Framework for Language Generation Evaluation

On the Use of Linguistic Features for the Evaluation of Generative Dialogue Systems

Reflexion: Language Agents with Verbal Reinforcement Learning

MIRROR: A Novel Approach for the Automated Evaluation of Open-Ended Question Generation

On the Evaluation of Machine-Generated Reports

Self-Reflection Outcome is Sensitive to Prompt Construction

MetaReflection: Learning Instructions for Language Agents using Past Reflections

MATEval: A Multi-Agent Discussion Framework for Advancing Open-Ended Text Evaluation

Reflection-Bench: probing AI intelligence with reflection

How NOT To Evaluate Your Dialogue System: An Empirical Study of Unsupervised Evaluation Metrics for Dialogue Response Generation

DUAL-REFLECT: Enhancing Large Language Models for Reflective Translation through Dual Learning Feedback Mechanisms

Generative AI as a Tool for Enhancing Reflective Learning in Students

LLMs as Narcissistic Evaluators: When Ego Inflates Evaluation Scores