Abstract:Assessing the quality of outputs generated by generative models, such as large language models and vision language models, presents notable challenges. Traditional methods for evaluation typically rely on either human assessments, which are resource-intensive, or automatic metrics that often show a low correlation with human judgment. Another common approach is to use deep learning systems, which not only consume a substantial amount of compute and time but also require extensive training data. In this study, we introduce a tuning-free framework called ReFeR, designed to evaluate generative outputs, including both text and images, by leveraging a 2-level hierarchy of LLMs and VLMs themselves. We rigorously evaluate our framework, ReFeR, across four diverse evaluation tasks. The framework not only improves the accuracy of these evaluations, surpassing previous benchmarks but also generates constructive feedback. Interestingly, the framework is also applicable to reasoning tasks. Experiments on four reasoning tasks demonstrate superior collective reasoning abilities of the framework. We present two variants of the framework: ReFeR-Turbo, optimized for accelerated performance, and ReFeR-Lite, offering a more cost-effective solution. ReFeR-Lite is $\sim7.7\times$ more efficient while being comparably accurate to ReFeR-Turbo. We make code, data and PIP package publicly available. See this PIP URL <a class="link-external link-https" href="https://pypi.org/project/refer-agents/" rel="external noopener nofollow">this https URL</a> and this Git URL <a class="link-external link-https" href="https://github.com/yaswanth-iitkgp/ReFeR_Code" rel="external noopener nofollow">this https URL</a> .

REFeREE: A REference-FREE Model-Based Metric for Text Simplification

Mitigating the Impact of Reference Quality on Evaluation of Summarization Systems with Reference-Free Metrics

Evaluating Document Simplification: On the Importance of Separately Assessing Simplicity and Meaning Preservation

CTRLEval: an Unsupervised Reference-Free Metric for Evaluating Controlled Text Generation

DocAsRef: an Empirical Study on Repurposing Reference-based Summary Quality Metrics As Reference-free Metrics

Towards a Comprehensive Metric for Evaluating Text Simplification Systems

Resource-Efficient Reference-Free Evaluation of Audio Captions

Towards Interpretable and Efficient Automatic Reference-Based Summarization Evaluation

NoRefER: a Referenceless Quality Metric for Automatic Speech Recognition via Semi-Supervised Language Model Fine-Tuning with Contrastive Learning

REAM$\sharp$: An Enhancement Approach to Reference-based Evaluation Metrics for Open-domain Dialog Generation

Is Reference Necessary in the Evaluation of NLG Systems? When and Where?

On the Limitations of Reference-Free Evaluations of Generated Text

Reference-based Metrics Disprove Themselves in Question Generation

ContextRef: Evaluating Referenceless Metrics For Image Description Generation

CERET: Cost-Effective Extrinsic Refinement for Text Generation

From Model-centered to Human-Centered: Revision Distance as a Metric for Text Evaluation in LLMs-based Applications

A Reference-less Quality Metric for Automatic Speech Recognition via Contrastive-Learning of a Multi-Language Model with Self-Supervision

Not All Metrics Are Guilty: Improving NLG Evaluation by Diversifying References

Rate, Explain and Cite (REC): Enhanced Explanation and Attribution in Automatic Evaluation by Large Language Models

ReFeR: Improving Evaluation and Reasoning through Hierarchy of Models

FLEUR: An Explainable Reference-Free Evaluation Metric for Image Captioning Using a Large Multimodal Model