Abstract:Most pre-trained learning systems are known to suffer from bias, which typically emerges from the data, the model, or both. Measuring and quantifying bias and its sources is a challenging task and has been extensively studied in image captioning. Despite the significant effort in this direction, we observed that existing metrics lack consistency in the inclusion of the visual signal. In this paper, we introduce a new bias assessment metric, dubbed $ImageCaptioner^2$, for image captioning. Instead of measuring the absolute bias in the model or the data, $ImageCaptioner^2$ pay more attention to the bias introduced by the model w.r.t the data bias, termed bias amplification. Unlike the existing methods, which only evaluate the image captioning algorithms based on the generated captions only, $ImageCaptioner^2$ incorporates the image while measuring the bias. In addition, we design a formulation for measuring the bias of generated captions as prompt-based image captioning instead of using language classifiers. Finally, we apply our $ImageCaptioner^2$ metric across 11 different image captioning architectures on three different datasets, i.e., MS-COCO caption dataset, Artemis V1, and Artemis V2, and on three different protected attributes, i.e., gender, race, and emotions. Consequently, we verify the effectiveness of our $ImageCaptioner^2$ metric by proposing AnonymousBench, which is a novel human evaluation paradigm for bias metrics. Our metric shows significant superiority over the recent bias metric; LIC, in terms of human alignment, where the correlation scores are 80% and 54% for our metric and LIC, respectively. The code is available at <a class="link-external link-https" href="https://eslambakr.github.io/imagecaptioner2.github.io/" rel="external noopener nofollow">this https URL</a>.

Mitigating Dataset Bias in Image Captioning Through Clip Confounder-Free Captioning Network

Deconfounded Image Captioning: A Causal Retrospect

Show, Deconfound and Tell: Image Captioning with Causal Inference

A Survey on Causal Inference in Image Captioning

Dependent Multi-Task Learning with Causal Intervention for Image Captioning.

Mitigating Gender Bias in Captioning Systems

ImageCaptioner$^2$: Image Captioner for Image Captioning Bias Amplification Assessment

Adaptive Syncretic Attention for Constrained Image Captioning

See or Guess: Counterfactually Regularized Image Captioning

Fine-grained Image Captioning with CLIP Reward

Controllable Dense Captioner with Multimodal Embedding Bridging

Contour-Augmented Concept Prediction Network for Image Captioning

Relation-aware Multi-pass Comparison Deconfounded Network for Change Captioning

Improving Multimodal Datasets with Image Captioning

DecoupleCLIP: A Novel Cross-Modality Decouple Model for Painting Captioning

Towards Deconfounded Image-Text Matching with Causal Inference

Contextual Debiasing for Visual Recognition with Causal Mechanisms

Image Captioning Based on Adaptive Balancing Loss.

Improving Image Captioning with Better Use of Caption

Concrete Image Captioning by Integrating Content Sensitive and Global Discriminative Objective.

OFCap:Object-aware Fusion for Image Captioning