Abstract:There has been a growing interest in developing multimodal machine translation (MMT) systems that enhance neural machine translation (NMT) with visual knowledge. This problem setup involves using images as auxiliary information during training, and more recently, eliminating their use during inference. Towards this end, previous works face a challenge in training powerful MMT models from scratch due to the scarcity of annotated multilingual vision-language data, especially for low-resource languages. Simultaneously, there has been an influx of multilingual pre-trained models for NMT and multimodal pre-trained models for vision-language tasks, primarily in English, which have shown exceptional generalisation ability. However, these are not directly applicable to MMT since they do not provide aligned multimodal multilingual features for generative tasks. To alleviate this issue, instead of designing complex modules for MMT, we propose CLIPTrans, which simply adapts the independently pre-trained multimodal M-CLIP and the multilingual mBART. In order to align their embedding spaces, mBART is conditioned on the M-CLIP features by a prefix sequence generated through a lightweight mapping network. We train this in a two-stage pipeline which warms up the model with image captioning before the actual translation task. Through experiments, we demonstrate the merits of this framework and consequently push forward the state-of-the-art across standard benchmarks by an average of +2.67 BLEU. The code can be found at <a class="link-external link-http" href="http://www.github.com/devaansh100/CLIPTrans" rel="external noopener nofollow">this http URL</a>.

MT6: Multilingual Pretrained Text-to-Text Transformer with Translation Pairs

mLongT5: A Multilingual and Efficient Text-To-Text Transformer for Longer Sequences

XLM-T: Scaling up Multilingual Machine Translation with Pretrained Cross-lingual Transformer Encoders

Zero-shot Cross-lingual Transfer of Neural Machine Translation with Multilingual Pretrained Encoders

Towards Making the Most of Multilingual Pretraining for Zero-Shot Neural Machine Translation

Towards Making the Most of Cross-Lingual Transfer for Zero-Shot Neural Machine Translation

Better Simultaneous Translation with Monotonic Knowledge Distillation.

Make the Blind Translator See The World: A Novel Transfer Learning Solution for Multimodal Machine Translation.

Improving End-to-End Text Image Translation From the Auxiliary Text Translation Task

Exploring Text-to-Text Transformers for English to Hinglish Machine Translation with Synthetic Code-Mixing

Improving Speech Translation by Understanding and Learning from the Auxiliary Text Translation Task

Crosslingual Generalization through Multitask Finetuning

Multilingual Multimodal Learning with Machine Translated Text

A multitask co-training framework for improving speech translation by leveraging speech recognition and machine translation tasks

HW-TSC's Participation in the WMT 2020 News Translation Shared Task.

Multi-Hop Transformer for Document-Level Machine Translation

Enhanced Pre-Trained Transformer with Aligned Attention Map for Text Matching

CLIPTrans: Transferring Visual Knowledge with Pre-trained Models for Multimodal Machine Translation

Multilingual Translation with Extensible Multilingual Pretraining and Finetuning

Unsupervised Cross-lingual Adaptation for Sequence Tagging and Beyond