Abstract:Most State-Of-The-Art (SOTA) Neural Machine Translation (NMT) systems today achieve outstanding results based only on large parallel corpora. The large-scale parallel corpora for high-resource languages is easily obtainable. However, the translation quality of NMT for morphologically rich languages is still unsatisfactory, mainly because of the data sparsity problem encountered in Low-Resource Languages (LRLs). In the low-resource NMT paradigm, Transfer Learning (TL) has been developed into one of the most efficient methods. It is difficult to train the model on high-resource languages to include the information in both parent and child models, as well as the initially trained model that only contains the lexicon features and word embeddings of the parent model instead of the child languages feature. In this work, we aim to address this issue by proposing the language-independent Hybrid Transfer Learning (HTL) method for LRLs by sharing lexicon embedding between parent and child languages without leveraging back translation or manually injecting noises. First, we train the High-Resource Languages (HRLs) as the parent model with its vocabularies. Then, we combine the parent and child language pairs using the oversampling method to train the hybrid model initialized by the previously parent model. Finally, we fine-tune the morphologically rich child model using a hybrid model. Besides, we explore some exciting discoveries on the original TL approach. Experimental results show that our model consistently outperforms five SOTA methods in two languages Azerbaijani (Az) and Uzbek (Uz). Meanwhile, our approach is practical and significantly better, achieving improvements of up to 4.94 and 4.84 BLEU points for low-resource child languages Az ! Zh and Uz ! Zh, respectively.

An Optimized Regularization Method to Enhance Low-Resource MT

Wordreg: Mitigating the Gap Between Training and Inference with Worst-Case Drop Regularization

Joint Dropout: Improving Generalizability in Low-Resource Neural Machine Translation through Phrase Pair Variables

Fixing MoE Over-Fitting on Low-Resource Languages in Multilingual Machine Translation

R-Drop: Regularized Dropout for Neural Networks.

Optimizing Deep Transformers for Chinese-Thai Low-Resource Translation

Handling Syntactic Divergence in Low-resource Machine Translation

Randomness Regularization with Simple Consistency Training for Neural Networks

Bi-Drop: Enhancing Fine-tuning Generalization Via Synchronous Sub-Net Estimation and Optimization

Low-Resource Machine Translation Training Curriculum Fit for Low-Resource Languages

DropDim: A Regularization Method for Transformer Networks

UniDrop: A Simple Yet Effective Technique to Improve Transformer Without Extra Cost.

Rethinking the Exploitation of Monolingual Data for Low-Resource Neural Machine Translation

Layer-wise Regularized Dropout for Neural Language Models

Improving Multilingual Translation by Representation and Gradient Regularization

TLM: Token-Level Masking for Transformers

Investigating the Synergistic Effects of Dropout and Residual Connections on Language Model Training

A Cooperative Lightweight Translation Algorithm Combined with Sparse-ReLU

A Survey on Low-Resource Neural Machine Translation

Regularizing neural networks with adaptive local drop

Enriching the Transfer Learning with Pre-Trained Lexicon Embedding for Low-Resource Neural Machine Translation