Sequence to Sequence Mixture Model for Diverse Machine Translation

Xuanli He,Gholamreza Haffari,Mohammad Norouzi

DOI: https://doi.org/10.48550/arXiv.1810.07391

2018-10-17

Abstract:Sequence to sequence (SEQ2SEQ) models often lack diversity in their generated translations. This can be attributed to the limitation of SEQ2SEQ models in capturing lexical and syntactic variations in a parallel corpus resulting from different styles, genres, topics, or ambiguity of the translation process. In this paper, we develop a novel sequence to sequence mixture (S2SMIX) model that improves both translation diversity and quality by adopting a committee of specialized translation models rather than a single translation model. Each mixture component selects its own training dataset via optimization of the marginal loglikelihood, which leads to a soft clustering of the parallel corpus. Experiments on four language pairs demonstrate the superiority of our mixture model compared to a SEQ2SEQ baseline with standard or diversity-boosted beam search. Our mixture model uses negligible additional parameters and incurs no extra computation cost during decoding.

Computation and Language

What problem does this paper attempt to address?

The problem this paper attempts to address is the lack of diversity in translations generated by neural sequence-to-sequence (SEQ2SEQ) models. Specifically, existing SEQ2SEQ models have limitations in capturing lexical and syntactic variations in parallel corpora, which may arise from different styles, genres, topics, or inherent ambiguities in the translation process. This results in generated translations often being very similar, even when using different decoding algorithms such as beam search. Therefore, the paper proposes a new sequence-to-sequence mixture (S2SMIX) model to improve translation diversity and quality by employing a set of specialized translation models rather than a single model. Each mixture component selects its own training dataset by optimizing the marginal log-likelihood, thereby achieving soft clustering of the parallel corpus. Experimental results show that this mixture model outperforms the standard or diversity-enhanced beam search SEQ2SEQ baseline models across 4 language pairs, with almost no additional parameters and no extra computational cost during decoding.