Abstract:The main challenge in the field of unsupervised machine translation (UMT) is to associate source-target sentences in the latent space. As people who speak different languages share biologically similar visual systems, various unsupervised multi-modal machine translation (UMMT) models have been proposed to improve the performances of UMT by employing visual contents in natural images to facilitate alignment. Commonly, relation information is the important semantic in a sentence. Compared with images, videos can better present the interactions between objects and the ways in which an object transforms over time. However, current state-of-the-art methods only explore scene-level or object-level information from images without explicitly modeling objects relation; thus, they are sensitive to spurious correlations, which poses a new challenge for UMMT models. In this paper, we employ a spatial-temporal graph obtained from videos to exploit object interactions in space and time for disambiguation purposes and to promote latent space alignment in UMMT. Our model employs multi-modal back-translation and features pseudo-visual pivoting, in which we learn a shared multilingual visual-semantic embedding space and incorporate visually pivoted captioning as additional weak supervision. Experimental results on the VATEX Translation 2020 and HowToWorld datasets validate the translation capabilities of our model on both sentence-level and word-level and generalizes well when videos are not available during the testing phase.

Unsupervised Multimodal Machine Translation for Low-Resource Distant Language Pairs

Multimodal Image-to-Image Translation via Mutual Information Estimation and Maximization

Towards Zero-Shot Multimodal Machine Translation

Supervised Visual Attention for Simultaneous Multimodal Machine Translation

Machine Translation With Weakly Paired Bilingual Documents

HybridVocab: Towards Multi-Modal Machine Translation Via Multi-Aspect Alignment

Video Pivoting Unsupervised Multi-Modal Machine Translation

Enhancing Neural Machine Translation with Dual-Side Multimodal Awareness

Machine Translation with Weakly Paired Documents

Effective Self-Mining of In-Context Examples for Unsupervised Machine Translation with LLMs

DAS-CL: Towards Multimodal Machine Translation Via Dual-Level Asymmetric Contrastive Learning

A Visual Attention Grounding Neural Model for Multimodal Machine Translation

Multilingual Multimodal Learning with Machine Translated Text

Good for Misconceived Reasons: An Empirical Revisiting on the Need for Visual Context in Multimodal Machine Translation

Unified Model Learning for Various Neural Machine Translation

Exploring the Necessity of Visual Modality in Multimodal Machine Translation using Authentic Datasets

3AM: An Ambiguity-Aware Multi-Modal Machine Translation Dataset

Adding Visual Information to Improve Multimodal Machine Translation for Low-Resource Language

MURAL: Multimodal, Multitask Retrieval Across Languages

Tackling Ambiguity with Images: Improved Multimodal Machine Translation and Contrastive Evaluation

Beyond English-Centric Multilingual Machine Translation