Abstract:A good evaluation framework should evaluate multimodal machine translation (MMT) models by measuring 1) their use of visual information to aid in the translation task and 2) their ability to translate complex sentences such as done for text-only machine translation. However, most current work in MMT is evaluated against the Multi30k testing sets, which do not measure these properties. Namely, the use of visual information by the MMT model cannot be shown directly from the Multi30k test set results and the sentences in Multi30k are are image captions, i.e., short, descriptive sentences, as opposed to complex sentences that typical text-only machine translation models are evaluated against.

What problem does this paper attempt to address?

The paper primarily focuses on the evaluation methods in the field of Multimodal Machine Translation (MMT), particularly addressing the shortcomings of existing evaluation frameworks in measuring model performance. Specifically, the paper highlights the limitations of the Multi30k dataset, which most current MMT research relies on for evaluation: 1. **Difficulty in directly evaluating the use of visual information**: Existing evaluation methods cannot directly prove whether and how MMT models effectively utilize image information to assist in the translation task. 2. **Insufficient sentence complexity**: The sentences in the Multi30k dataset are mainly image descriptions, usually short and structurally simple, unlike the complex sentences commonly found in text machine translation. 3. **Limitations of the dataset itself**: Many sentences in the Multi30k dataset can be correctly translated even without images because the sentences themselves are not ambiguous. Based on the above issues, the paper proposes a new evaluation framework aimed at better assessing two key aspects of MMT models: - **Ability to utilize visual information**: Using the CoMMuTE evaluation framework to measure whether the model can effectively use image information to resolve ambiguities in language. - **Ability to handle complex sentences**: Using the WMT news translation task test set to evaluate the model's ability to translate complex sentences. Additionally, the paper evaluates the performance of two Transformer-based MMT models (Gated Fusion and RMMT) under the newly proposed evaluation framework and compares them with a text-only translation model (FAIR-WMT19). The results show that although these two MMT models perform well on the Multi30k test set, their performance significantly drops when handling the more complex WMT news translation task test set. This indicates that the currently trained MMT models may face challenges in practical applications, especially when dealing with complex sentences. Therefore, the paper emphasizes the importance of improving evaluation methods and suggests that future research directions should include designing MMT models that can better handle the fundamental tasks of text translation.

The Case for Evaluating Multimodal Translation Models on Text Datasets

Adding Multimodal Capabilities to a Text-only Translation Model

Vision Matters When It Should: Sanity Checking Multimodal Machine Translation Models

Good for Misconceived Reasons: An Empirical Revisiting on the Need for Visual Context in Multimodal Machine Translation

Exploring the Necessity of Visual Modality in Multimodal Machine Translation using Authentic Datasets

Tackling Ambiguity with Images: Improved Multimodal Machine Translation and Contrastive Evaluation

Multilingual Multimodal Learning with Machine Translated Text

HybridVocab: Towards Multi-Modal Machine Translation Via Multi-Aspect Alignment

Probing the Need for Visual Context in Multimodal Machine Translation

MM-BigBench: Evaluating Multimodal Models on Multimodal Content Comprehension Tasks

Increasing Visual Awareness in Multimodal Neural Machine Translation from an Information Theoretic Perspective

MM-InstructEval: Zero-Shot Evaluation of (Multimodal) Large Language Models on Multimodal Reasoning Tasks

MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilities

Supervised Visual Attention for Simultaneous Multimodal Machine Translation

MM-Vet v2: A Challenging Benchmark to Evaluate Large Multimodal Models for Integrated Capabilities

Contrastive Learning Based Visual Representation Enhancement for Multimodal Machine Translation

MMR: Evaluating Reading Ability of Large Multimodal Models

Multimodal Transformer For Multimodal Machine Translation

Towards Zero-Shot Multimodal Machine Translation

A Multi-task Learning Framework for Evaluating Machine Translation of Emotion-loaded User-generated Content

Multi-Dimensional Machine Translation Evaluation: Model Evaluation and Resource for Korean