Abstract:As an extension of machine translation, the primary objective of multi-modal machine translation is to optimize the utilization of visual information. Technically, image information is integrated into multi-modal fusion and alignment as an auxiliary modality through concepts or latent semantics, which are typically based on the Transformer framework. However, current approaches often ignore one modality to design numerous handcrafted features (e.g. visual concept extraction) and require training of all parameters in their framework. Therefore, it is worthwhile to explore multi-modal concepts or features to enhance performance and an efficient approach to incorporate visual information with minimal cost. Meanwhile, with the development of multi-modal large language models (MLLMs), they are faced with the visual hallucination issue of compromising performance, despite their powerful capabilities. Inspired by pioneering techniques in the multi-modal field, such as prompt learning and MLLMs, this paper innovatively explores the possibility of applying multi-modal prompt learning to this multi-modal machine translation task. Our framework offers three key advantages: it establishes a robust connection between visual concepts and translation processes, requires a minimum of 1.46M parameters for training, and can be seamlessly integrated into any existing framework by retrieving a multi-modal dictionary. Specifically, we propose two prompt-guided strategies: a learnable prompt-refined module and a heuristic prompt-refined module. Among them, the learnable strategy utilizes off-the-shelf pre-trained models, while the heuristic strategy constrains the hallucination problem via concept retrieval. Our experiments on two real-world benchmark datasets demonstrate that our proposed method outperforms all competitors.

DAS-CL: Towards Multimodal Machine Translation Via Dual-Level Asymmetric Contrastive Learning

Contrastive Learning Based Visual Representation Enhancement for Multimodal Machine Translation

HybridVocab: Towards Multi-Modal Machine Translation Via Multi-Aspect Alignment

Enhancing Neural Machine Translation with Dual-Side Multimodal Awareness

Dynamic Context-guided Capsule Network for Multimodal Machine Translation

Contrastive Adversarial Training for Multi-modal Machine Translation

Feature-level Incongruence Reduction for Multimodal Translation

Dual-level interactive multimodal-mixup encoder for multi-modal neural machine translation

On the Generalization of Multi-modal Contrastive Learning

Tackling Ambiguity with Images: Improved Multimodal Machine Translation and Contrastive Evaluation

Dual-Level Cross-Modal Contrastive Clustering

Soul-Mix: Enhancing Multimodal Machine Translation with Manifold Mixup

Multimodal Neural Machine Translation for Mongolian to Chinese

Multilingual Multimodal Learning with Machine Translated Text

RetrievalMMT: Retrieval-Constrained Multi-Modal Prompt Learning for Multi-Modal Machine Translation

Multi-modal Neural Machine Translation with Deep Semantic Interactions.

Multiple Contrastive Learning for Multimodal Sentiment Analysis

Good for Misconceived Reasons: An Empirical Revisiting on the Need for Visual Context in Multimodal Machine Translation

Towards Zero-Shot Multimodal Machine Translation

CLIP-enhanced multimodal machine translation: integrating visual and label features with transformer fusion

3AM: An Ambiguity-Aware Multi-Modal Machine Translation Dataset