Abstract:Machine translation has received significant attention in the field of natural language processing not only because of its challenges but also due to the translation needs that arise in the daily life of modern people. In this study, we design a new machine translation model named X-Transformer, which refines the original Transformer model regarding three aspects. First, the model parameter of the encoder is compressed. Second, the encoder structure is modified by adopting two layers of the self-attention mechanism consecutively and reducing the point-wise feed forward layer to help the model understand the semantic structure of sentences precisely. Third, we streamline the decoder model size, while maintaining the accuracy. Through experiments, we demonstrate that having a large number of decoder layers not only affects the performance of the translation model but also increases the inference time. The X-Transformer reaches the state-of-the-art result of 46.63 and 55.63 points in the BiLingual Evaluation Understudy (BLEU) metric of the World Machine Translation (WMT), from 2014, using the English–German and English–French translation corpora, thus outperforming the Transformer model with 19 and 18 BLEU points, respectively. The X-Transformer significantly reduces the training time to only 1/3 times that of the Transformer. In addition, the heat maps of the X-Transformer reach token-level precision (i.e., token-to-token attention), while the Transformer model remains at the sentence level (i.e., token-to-sentence attention).

Character-Level Translation with Self-attention

X-Transformer: A Machine Translation Model Enhanced by the Self-Attention Mechanism

Character-based Neural Machine Translation

Self-Attention and Dynamic Convolution Hybrid Model for Neural Machine Translation

An Efficient Character-Level Neural Machine Translation.

A Character-Level Decoder without Explicit Segmentation for Neural Machine Translation

Understanding Pure Character-Based Neural Machine Translation: The Case of Translating Finnish into English

Low-Rank and Locality Constrained Self-Attention for Sequence Modeling.

Character-level NMT and language similarity

Why Self-Attention? A Targeted Evaluation of Neural Machine Translation Architectures

Character-level Convolutional Networks for Text Classification

A Character-Aware Encoder for Neural Machine Translation.

Neural Machine Translation with Recurrent Attention Modeling

Empirical Study on Character Level Neural Network Classifier for Chinese Text.

Training Deeper Neural Machine Translation Models with Transparent Attention

A Deep Investigation of RNN and Self-attention for the Cyrillic-Traditional Mongolian Bidirectional Conversion

Combining Character and Word Information in Neural Machine Translation Using a Multi-Level Attention.

Self-attention based end-to-end Hindi-English Neural Machine Translation

Is Attention All What You Need? -- An Empirical Investigation on Convolution-Based Active Memory and Self-Attention

Attention is All you Need

Are Character-level Translations Worth the Wait? Comparing ByT5 and mT5 for Machine Translation