Abstract:With significant efforts in recent studies, LLM-as-a-Judge has become a cost-effective alternative to human evaluation for assessing the text generation quality in a wide range of tasks. However, there still remains a reliability gap between LLM-as-a-Judge and human evaluation. One important reason is the lack of guided oracles in the evaluation process. Motivated by the role of reference pervasively used in classic text evaluation, we introduce RevisEval, a novel text generation evaluation paradigm via the response-adapted references. RevisEval is driven by the key observation that an ideal reference should maintain the necessary relevance to the response to be evaluated. Specifically, RevisEval leverages the text revision capabilities of large language models (LLMs) to adaptively revise the response, then treat the revised text as the reference (response-adapted reference) for the subsequent evaluation. Extensive experiments demonstrate that RevisEval outperforms traditional reference-free and reference-based evaluation paradigms that use LLM-as-a-Judge across NLG tasks and open-ended instruction-following tasks. More importantly, our response-adapted references can further boost the classical text metrics, e.g., BLEU and BERTScore, compared to traditional references and even rival the LLM-as-a-Judge. A detailed analysis is also conducted to confirm RevisEval's effectiveness in bias reduction, the impact of inference cost, and reference relevance.

The jsRealB Text Realizer: Organization and Use Cases -- Revised version

ReALM: Reference Resolution As Language Modeling

Intelligent Voice Prosthesis: Converting Icons into Natural Language Sentences

ARTIST: ARTificial Intelligence for Simplified Text

Natural Language Processing RELIES on Linguistics

Reagent: Converting Ordinary Webpages into Interactive Software Agents

RealitySummary: Exploring On-Demand Mixed Reality Text Summarization and Question Answering using Large Language Models

Using the LARA platform to crowdsource a multilingual, multimodal Little Prince

Helping Domain Experts Build Speech Translation Systems

Step-by-Step: Separating Planning from Realization in Neural Data-to-Text Generation

jsCoq: Towards Hybrid Theorem Proving Interfaces

Speech ReaLLM -- Real-time Streaming Speech Recognition with Multimodal LLMs by Teaching the Flow of Time

General Natural Language Real Understanding'S Realization

Neural Text Generation from Rich Semantic Representations

ReNeLiB: Real-time Neural Listening Behavior Generation for Socially Interactive Agents

Dependency Based Chinese Sentence Realization

RevisEval: Improving LLM-as-a-Judge via Response-Adapted References

Refining the state-of-the-art in Machine Translation, optimizing NMT for the JA <-> EN language pair by leveraging personal domain expertise

Evaluating Factual Consistency of Texts with Semantic Role Labeling

LLMR: Real-time Prompting of Interactive Worlds using Large Language Models

Automatic semantic role labeling on non-revised syntactic trees of journalistic texts