Abstract:Multi-modal machine translation (MMT) aims to augment the linguistic machine translation frameworks by incorporating aligned vision information. As the core research challenge for MMT, how to fuse the image information and further align it with the bilingual data remains critical. Existing works have either focused on a methodological alignment in the space of bilingual text or emphasized the combination of the one-sided text and given image. In this work, we entertain the possibility of a triplet alignment, among the source and target text together with the image instance. In particular, we propose Multi-aspect AlignmenT (MAT) model that augments the MMT tasks to three sub-tasks --- namely cross-language translation alignment, cross-modal captioning alignment and multi-modal hybrid alignment tasks. Core to this model consists of a hybrid vocabulary which compiles the visually depictable entity (nouns) occurrence on both sides of the text as well as the detected object labels appearing in the images. Through this sub-task, we postulate that MAT manages to further align the modalities by casting three instances into a shared domain, as compared against previously proposed methods. Extensive experiments and analyses demonstrate the superiority of our approaches, which achieve several state-of-the-art results on two benchmark datasets of the MMT task.

A Hybrid Approach To Skeleton-Based Translation

A Hybrid Approach to Chinese-English Machine Translation

A Hybrid Approach For Hindi-English Machine Translation

HybridVocab: Towards Multi-Modal Machine Translation Via Multi-Aspect Alignment

Exploring Hybrid Character-Words Representational Unit in Classical-to-Modern Chinese Machine Translation

An Improved Hierarchical Phrase Based Machine Translation Model

On Efficient Coupling of ASR and SMT for Speech Translation

Better Simultaneous Translation with Monotonic Knowledge Distillation.

Vietnamese to Chinese Machine Translation Via Chinese Character As Pivot.

Towards Integrated Machine Translation Using Structural Alignment From Syntax-Augmented Synchronous Parsing

Improvement of Korean-Chinese Machine Translation Based on Complex Sentence Deconstruction

Hybrid Decoding: Decoding with Partial Hypotheses Combination over Multiple SMT Systems

A Hybrid Model for Enhancing Lexical Statistical Machine Translation (SMT)

Sentence-Level Paraphrasing for Machine Translation System Combination.

Prior Derivation Models For Formally Syntax-Based Translation Using Linguistically Syntactic Parsing and Tree Kernels.

Train the Machine with What It Can Learn

Neural System Combination For Machine Translation

Enriching SMT Training Data Via Paraphrasing.

A Tree-to-tree Alignment-Based Model for Statistical Machine Translation

Meta-Structure Transformation Model for Statistical Machine Translation.

Head-modifier relation based non-lexical reordering model for phrase-based translation