Abstract:Image captioning evaluation metrics can be divided into two categories, reference-based metrics and reference-free metrics. However, reference-based approaches may struggle to evaluate descriptive captions with abundant visual details produced by advanced multimodal large language models, due to their heavy reliance on limited human-annotated references. In contrast, previous reference-free metrics have been proven effective via CLIP cross-modality similarity. Nonetheless, CLIP-based metrics, constrained by their solution of global image-text compatibility, often have a deficiency in detecting local textual hallucinations and are insensitive to small visual objects. Besides, their single-scale designs are unable to provide an interpretable evaluation process such as pinpointing the position of caption mistakes and identifying visual regions that have not been described. To move forward, we propose a novel reference-free metric for image captioning evaluation, dubbed Hierarchical Image Captioning Evaluation Score (HICE-S). By detecting local visual regions and textual phrases, HICE-S builds an interpretable hierarchical scoring mechanism, breaking through the barriers of the single-scale structure of existing reference-free metrics. Comprehensive experiments indicate that our proposed metric achieves the SOTA performance on several benchmarks, outperforming existing reference-free metrics like CLIP-S and PAC-S, and reference-based metrics like METEOR and CIDEr. Moreover, several case studies reveal that the assessment process of HICE-S on detailed captions closely resembles interpretable human judgments.Our code is available at <a class="link-external link-https" href="https://github.com/joeyz0z/HICE" rel="external noopener nofollow">this https URL</a>.

CogME: A Cognition-Inspired Multi-Dimensional Evaluation Metric for Story Understanding

MiCEval: Unveiling Multimodal Chain of Thought's Quality via Image Description and Reasoning Steps

What Makes a Good Story and How Can We Measure It? A Comprehensive Survey of Story Evaluation

Measuring Psychological Depth in Language Models

Learning to Evaluate Image Captioning

OpenMEVA: A Benchmark for Evaluating Open-ended Story Generation Metrics

Dialogues Are Not Just Text: Modeling Cognition for Dialogue Coherence Evaluation

Enhancing human-like multimodal reasoning: a new challenging dataset and comprehensive framework

Understanding ME? Multimodal Evaluation for Fine-grained Visual Commonsense

MME-CRS: Multi-Metric Evaluation Based on Correlation Re-Scaling for Evaluating Open-Domain Dialogue

Enhancing Human-like Multi-Modal Reasoning: A New Challenging Dataset and Comprehensive Framework

RoViST:Learning Robust Metrics for Visual Storytelling

Multi-Dimensional Machine Translation Evaluation: Model Evaluation and Resource for Korean

DramaQA: Character-Centered Video Story Understanding with Hierarchical QA

Towards a Unified Multi-Dimensional Evaluator for Text Generation.

CMMU: A Benchmark for Chinese Multi-modal Multi-type Question Understanding and Reasoning

HICEScore: A Hierarchical Metric for Image Captioning Evaluation

MulCogBench: A Multi-modal Cognitive Benchmark Dataset for Evaluating Chinese and English Computational Language Models

Hypergraph Multi-modal Large Language Model: Exploiting EEG and Eye-tracking Modalities to Evaluate Heterogeneous Responses for Video Understanding

The Inside Story: Towards Better Understanding of Machine Translation Neural Evaluation Metrics

MMMU-Pro: A More Robust Multi-discipline Multimodal Understanding Benchmark