Abstract:Current multimodal neural machine translation (MNMT) approaches primarily focus on ensuring consistency between visual annotations and the source language, often overlooking the broader aspect of multimodal coherence, including target–visual and bilingual–visual alignment. In this paper, we propose a novel approach that effectively leverages target–visual consistency (TVC) and bilingual–visual consistency (BiVC) to improve MNMT performance. Our method leverages visual annotations depicting concepts across bilingual parallel sentences to enhance multimodal coherence in translation. We exploit target–visual harmony by extracting contextual cues from visual annotations during auto-regressive decoding, incorporating vital future context to improve target sentence representation. Additionally, we introduce a consistency loss promoting semantic congruence between bilingual sentence pairs and their visual annotations, fostering a tighter integration of textual and visual modalities. Extensive experiments on diverse multimodal translation datasets empirically demonstrate our approach's effectiveness. This visually aware, data-driven framework opens exciting opportunities for intelligent learning, adaptive control, and robust distributed optimization of multi-agent systems in uncertain, complex environments. By seamlessly fusing multimodal data and machine learning, our method paves the way for novel control paradigms capable of effectively handling the dynamics and constraints of real-world multi-agent applications.

A Visual Attention Grounding Neural Model for Multimodal Machine Translation

Multimodal Image-to-Image Translation via Mutual Information Estimation and Maximization

Enhancing Neural Machine Translation with Dual-Side Multimodal Awareness

Visual Agreement Regularized Training for Multi-Modal Machine Translation

Increasing Visual Awareness in Multimodal Neural Machine Translation from an Information Theoretic Perspective

HybridVocab: Towards Multi-Modal Machine Translation Via Multi-Aspect Alignment

Good for Misconceived Reasons: An Empirical Revisiting on the Need for Visual Context in Multimodal Machine Translation

Exploring the Necessity of Visual Modality in Multimodal Machine Translation using Authentic Datasets

Bilingual–Visual Consistency for Multimodal Neural Machine Translation

Supervised Visual Attention for Simultaneous Multimodal Machine Translation

Imagination improves Multimodal Translation

Multimodal Unified Attention Networks for Vision-and-Language Interactions

Vision Matters When It Should: Sanity Checking Multimodal Machine Translation Models

Multimodal Neural Machine Translation with Search Engine Based Image Retrieval

Unpaired Multimodal Neural Machine Translation via Reinforcement Learning

Multilingual Multimodal Learning with Machine Translated Text

Visual Topic Semantic Enhanced Machine Translation for Multi-Modal Data Efficiency

Xmodel-VLM: A Simple Baseline for Multimodal Vision Language Model

Multi-grained visual pivot-guided multi-modal neural machine translation with text-aware cross-modal contrastive disentangling

UC2: Universal Cross-lingual Cross-modal Vision-and-Language Pre-training

Contrastive Learning Based Visual Representation Enhancement for Multimodal Machine Translation