Abstract:Capturing the underlying semantic relationships of sentences is helpful for machine translation. Variational neural machine translation approaches provide an effective way to model the uncertain underlying semantics in languages by introducing latent variables. Multitask learning is applied in multimodal machine translation to integrate multimodal data. However, these approaches usually lack a strong interpretation in utilizing out-of-text information in machine translation tasks. In this paper, we propose a novel architecture-free multimodal translation model, called variational multimodal machine translation (VMMT), under the variational framework which can model the uncertainty in languages caused by ambiguity through utilizing visual and textual information. In addition, the proposed model can eliminate the discrepancy between training and prediction in the existing variational translation models by constructing encoders only relying on source data. More importantly, the proposed multimodal translation model is designed as multitask learning in which the shared semantic representation for different modes is learned and the gap among semantic representation from various modes is reduced by incorporating additional constraints. Moreover, the information bottleneck theory is adopted in our variational encoder–decoder model, which helps the encoder to filter redundancy and the decoder to concentrate on useful information. Experiments on multimodal machine translation demonstrate that the proposed model is competitive.

Team RUC AI·M Technical Report at VMT Challenge 2020: Enhancing Neural Machine Translation with Multimodal Rewards

Mutual Information and Diverse Decoding Improve Neural Machine Translation.

Enhancing Neural Machine Translation with Dual-Side Multimodal Awareness

Increasing Visual Awareness in Multimodal Neural Machine Translation from an Information Theoretic Perspective

Bilingual–Visual Consistency for Multimodal Neural Machine Translation

A Visual Attention Grounding Neural Model for Multimodal Machine Translation

Unpaired Multimodal Neural Machine Translation via Reinforcement Learning

HybridVocab: Towards Multi-Modal Machine Translation Via Multi-Aspect Alignment

Vision Matters When It Should: Sanity Checking Multimodal Machine Translation Models

Feature-level Incongruence Reduction for Multimodal Translation

Good for Misconceived Reasons: An Empirical Revisiting on the Need for Visual Context in Multimodal Machine Translation

Multimodal Neural Machine Translation for Mongolian to Chinese

Multi-modal Neural Machine Translation with Deep Semantic Interactions.

Dual-level interactive multimodal-mixup encoder for multi-modal neural machine translation

Variational Multimodal Machine Translation with Underlying Semantic Alignment

Neural Machine Translation with Universal Visual Representation.

Contrastive Learning Based Visual Representation Enhancement for Multimodal Machine Translation

Improving Multilingual Translation by Representation and Gradient Regularization

Supervised Visual Attention for Simultaneous Multimodal Machine Translation

Incorporating Probing Signals into Multimodal Machine Translation Via Visual Question-Answering Pairs

Achieving Open Vocabulary Neural Machine Translation with Hybrid Word-Character Models