Abstract:A rising interest in the modality extension of foundation language models warrants discussion on the most effective, and efficient, multimodal training approach. This work focuses on neural machine translation (NMT) and proposes a joint multimodal training regime of Speech-LLM to include automatic speech translation (AST). We investigate two different foundation model architectures, decoder-only GPT and encoder-decoder T5, extended with Canary-1B's speech encoder. To handle joint multimodal training, we propose a novel training framework called EMMeTT. EMMeTT improves training efficiency with the following: balanced sampling across languages, datasets, and modalities; efficient sequential data iteration; and a novel 2D bucketing scheme for multimodal data, complemented by a batch size optimizer (OOMptimizer). We show that a multimodal training consistently helps with both architectures. Moreover, SALM-T5 trained with EMMeTT retains the original NMT capability while outperforming AST baselines on four-language subsets of FLORES and FLEURS. The resultant Multimodal Translation Model produces strong text and speech translation results at the same time.

LIUM-CVC Submissions for WMT18 Multimodal Translation Task

LIUM-CVC Submissions for WMT17 Multimodal Translation Task

LIUM Machine Translation Systems for WMT17 News Translation Task

UMONS Submission for WMT18 Multimodal Translation Task

NICT's Neural and Statistical Machine Translation Systems for the WMT18 News Translation Task

XMU Neural Machine Translation Systems for WMT 17.

SJTU-NICT's Supervised and Unsupervised Neural Machine Translation Systems for the WMT20 News Translation Task

The University of Edinburgh's Neural MT Systems for WMT17

HW-TSC's Submission to the CCMT 2024 Machine Translation Tasks

Findings of the Second Shared Task on Multimodal Machine Translation and Multilingual Image Description

EMMeTT: Efficient Multimodal Machine Translation Training

TSMind: Alibaba and Soochow University's Submission to the WMT22 Translation Suggestion Task

Facebook AI WMT21 News Translation Task Submission

Choose the Final Translation from NMT and LLM hypotheses Using MBR Decoding: HW-TSC's Submission to the WMT24 General MT Shared Task

Multilingual Machine Translation Systems from Microsoft for WMT21 Shared Task

Ensemble Sequence Level Training for Multimodal MT: OSU-Baidu WMT18 Multimodal Machine Translation System Report

The LMU Munich System for the WMT 2020 Unsupervised Machine Translation Shared Task

A Visual Attention Grounding Neural Model for Multimodal Machine Translation

Microsoft Research Asia's Systems for WMT19

QCRI Machine Translation Systems for IWSLT 16

NJUNLP’s Machine Translation System for CCMT-2020 Uighur $$\rightarrow $$ Chinese Translation Task