Abstract:Current multimodal neural machine translation (MNMT) approaches primarily focus on ensuring consistency between visual annotations and the source language, often overlooking the broader aspect of multimodal coherence, including target–visual and bilingual–visual alignment. In this paper, we propose a novel approach that effectively leverages target–visual consistency (TVC) and bilingual–visual consistency (BiVC) to improve MNMT performance. Our method leverages visual annotations depicting concepts across bilingual parallel sentences to enhance multimodal coherence in translation. We exploit target–visual harmony by extracting contextual cues from visual annotations during auto-regressive decoding, incorporating vital future context to improve target sentence representation. Additionally, we introduce a consistency loss promoting semantic congruence between bilingual sentence pairs and their visual annotations, fostering a tighter integration of textual and visual modalities. Extensive experiments on diverse multimodal translation datasets empirically demonstrate our approach's effectiveness. This visually aware, data-driven framework opens exciting opportunities for intelligent learning, adaptive control, and robust distributed optimization of multi-agent systems in uncertain, complex environments. By seamlessly fusing multimodal data and machine learning, our method paves the way for novel control paradigms capable of effectively handling the dynamics and constraints of real-world multi-agent applications.

Simultaneous Machine Translation with Visual Context

Supervised Visual Attention for Simultaneous Multimodal Machine Translation

Probing the Need for Visual Context in Multimodal Machine Translation

Context Consistency Between Training and Testing in Simultaneous Machine Translation.

Context Consistency between Training and Inference in Simultaneous Machine Translation

Better Simultaneous Translation with Monotonic Knowledge Distillation.

Good for Misconceived Reasons: An Empirical Revisiting on the Need for Visual Context in Multimodal Machine Translation

Enhancing Neural Machine Translation with Dual-Side Multimodal Awareness

Improving Simultaneous Machine Translation with Monolingual Data

HybridVocab: Towards Multi-Modal Machine Translation Via Multi-Aspect Alignment

Scene Graph as Pivoting: Inference-time Image-free Unsupervised Multimodal Machine Translation with Visual Scene Hallucination

Anticipation-Free Training for Simultaneous Machine Translation

Vision Matters When It Should: Sanity Checking Multimodal Machine Translation Models

Visual Topic Semantic Enhanced Machine Translation for Multi-Modal Data Efficiency

Anticipation-free Training for Simultaneous Translation

A Visual Attention Grounding Neural Model for Multimodal Machine Translation

It is Not as Good as You Think! Evaluating Simultaneous Machine Translation on Interpretation Data

Tackling Ambiguity with Images: Improved Multimodal Machine Translation and Contrastive Evaluation

DAS-CL: Towards Multimodal Machine Translation Via Dual-Level Asymmetric Contrastive Learning

Bilingual–Visual Consistency for Multimodal Neural Machine Translation

Exploring the Necessity of Visual Modality in Multimodal Machine Translation using Authentic Datasets