PROXYQA: An Alternative Framework for Evaluating Long-Form Text Generation with Large Language Models

Haochen Tan,Zhijiang Guo,Zhan Shi,Lu Xu,Zhili Liu,Yunlong Feng,Xiaoguang Li,Yasheng Wang,Lifeng Shang,Qun Liu,Linqi Song

2024-06-04

Abstract:Large Language Models (LLMs) have succeeded remarkably in understanding long-form contents. However, exploring their capability for generating long-form contents, such as reports and articles, has been relatively unexplored and inadequately assessed by existing benchmarks. The prevalent evaluation methods, which predominantly rely on crowdsourcing, are recognized for their labor-intensive nature and lack of efficiency, whereas automated metrics, such as the ROUGE score, demonstrate discordance with human judgment criteria. In this paper, we propose ProxyQA, an innovative framework dedicated to assessing long-text generation. ProxyQA comprises in-depth human-curated meta-questions spanning various domains, each accompanied by specific proxy-questions with pre-annotated answers. LLMs are tasked to generate extensive content in response to these meta-questions, by engaging an evaluator and incorporating the generated texts as contextual background, ProxyQA assesses the generated content's quality through the evaluator's accuracy in addressing the proxy-questions. We examine multiple LLMs, emphasizing ProxyQA's demanding nature as a high-quality assessment tool. Human evaluation demonstrates that the proxy-question method is notably self-consistent and aligns closely with human evaluative standards. The dataset and leaderboard is available at \url{<a class="link-external link-https" href="https://proxy-qa.com" rel="external noopener nofollow">this https URL</a>}.

Computation and Language,Artificial Intelligence

What problem does this paper attempt to address?

The problem that this paper attempts to solve is that the existing evaluation methods are deficient in generating long - text content. Specifically, current methods mainly rely on crowdsourcing evaluation, which is not only labor - intensive and inefficient, but also the automated metrics (such as ROUGE scores) are inconsistent with human judgment criteria. In addition, the existing benchmarks mainly focus on the ability of large - language models (LLMs) to understand long - form content, while relatively little research and evaluation have been done on their ability to generate long - form content. To make up for this deficiency, the author proposes PROXY QA, an innovative framework specifically designed to evaluate the quality of long - text generation. PROXY QA achieves this through a series of human - curated meta - questions, each of which is accompanied by a set of proxy - questions with pre - labeled answers. Large - language models need to generate detailed content based on these meta - questions, and then the quality of the generated content is evaluated by the accuracy of the evaluator answering the proxy - questions based on the generated content. This method aims to more accurately reflect the knowledge coverage and information volume of the generated content, thereby providing a higher - quality evaluation tool.

PROXYQA: An Alternative Framework for Evaluating Long-Form Text Generation with Large Language Models

HelloBench: Evaluating Long Text Generation Capabilities of Large Language Models

Rethinking Generative Large Language Model Evaluation for Semantic Comprehension

An Automatic and Cost-Efficient Peer-Review Framework for Language Generation Evaluation

Automated Evaluation of Personalized Text Generation using Large Language Models

Leave No Document Behind: Benchmarking Long-Context LLMs with Extended Multi-Doc QA

NovelQA: Benchmarking Question Answering on Documents Exceeding 200K Tokens

Automatic Large Language Model Evaluation Via Peer Review

RAG-QA Arena: Evaluating Domain Robustness for Long-form Retrieval Augmented Question Answering

Long-Span Question-Answering: Automatic Question Generation and QA-System Ranking via Side-by-Side Evaluation

Self-Boosting Large Language Models with Synthetic Preference Data

MATEval: A Multi-Agent Discussion Framework for Advancing Open-Ended Text Evaluation

PRE: A Peer Review Based Large Language Model Evaluator

Investigating Answerability of LLMs for Long-Form Question Answering

Eliciting Informative Text Evaluations with Large Language Models

Putting People in LLMs' Shoes: Generating Better Answers via Question Rewriter

RepEval: Effective Text Evaluation with LLM Representation

Leveraging Large Language Models for NLG Evaluation: Advances and Challenges

Collaborative Evaluation: Exploring the Synergy of Large Language Models and Humans for Open-ended Generation Evaluation

PPLqa: An Unsupervised Information-Theoretic Quality Metric for Comparing Generative Large Language Models

Large Language Models Are Active Critics in NLG Evaluation