Abstract:Recently, universal neural machine translation (NMT) with shared encoder-decoder gained good performance on zero-shot translation. Unlike universal NMT, jointly trained language-specific encoders-decoders aim to achieve universal representation across non-shared modules, each of which is for a language or language family. The non-shared architecture has the advantage of mitigating internal language competition, especially when the shared vocabulary and model parameters are restricted in their size. However, the performance of using multiple encoders and decoders on zero-shot translation still lags behind universal NMT. In this work, we study zero-shot translation using language-specific encoders-decoders. We propose to generalize the non-shared architecture and universal NMT by differentiating the Transformer layers between language-specific and interlingua. By selectively sharing parameters and applying cross-attentions, we explore maximizing the representation universality and realizing the best alignment of language-agnostic information. We also introduce a denoising auto-encoding (DAE) objective to jointly train the model with the translation task in a multi-task manner. Experiments on two public multilingual parallel datasets show that our proposed model achieves a competitive or better results than universal NMT and strong pivot baseline. Moreover, we experiment incrementally adding new language to the trained model by only updating the new model parameters. With this little effort, the zero-shot translation between this newly added language and existing languages achieves a comparable result with the model trained jointly from scratch on all languages.

Tied Transformers: Neural Machine Translation with Shared Encoder and Decoder

Mutual Information and Diverse Decoding Improve Neural Machine Translation.

Beyond Shared Vocabulary: Increasing Representational Word Similarities across Languages for Multilingual Machine Translation

Deep Transformer Modeling Via Grouping Skip Connection for Neural Machine Translation

Enhanced Neural Machine Translation by Joint Decoding with Word and POS-tagging Sequences.

Shared-Private Bilingual Word Embeddings for Neural Machine Translation

Transductive Ensemble Learning for Neural Machine Translation.

Toward Multilingual Neural Machine Translation with Universal Encoder and Decoder

Improving Language Transfer Capability of Decoder-only Architecture in Multilingual Neural Machine Translation

What Works and Doesn’t Work, A Deep Decoder for Neural Machine Translation

Effective Cross-lingual Transfer of Neural Machine Translation Models without Shared Vocabularies

Explicitly Modeling Word Translations in Neural Machine Translation

Transformer with Layer Fusion and Interaction

Balancing Cost and Benefit with Tied-Multi Transformers

Multi-channel Encoder for Neural Machine Translation

Layer-Wise Coordination Between Encoder and Decoder for Neural Machine Translation

Improving Zero-shot Neural Machine Translation on Language-specific Encoders-Decoders

Lattice-Based Transformer Encoder for Neural Machine Translation

Deep Recurrent Models with Fast-Forward Connections for Neural Machine Translation.

Multi-Unit Transformers for Neural Machine Translation

Joint-training on Symbiosis Networks for Deep Nueral Machine Translation models