Low-Resource Neural Machine Translation Based on Improved Reptile Meta-learning Method

Nier Wu,Hongxu Hou,Xiaoning Jia,Xin Chang,Haoran Li
DOI: https://doi.org/10.1007/978-981-16-7512-6_4
2021-01-01
Abstract:Multilingual transfer learning has been proved an effective method to solve the problem of low-resource neural machine translation (NMT). However, the global optimal parameters obtained through transfer learning can not effectively adapt to new tasks, which means the problem of local optimum will be caused when training the new task model. Although this problem can be alleviated by optimization-based meta-learning methods, but meta-parameters are determined by the second-order gradient term corresponding to the model parameters of a specific task, which consumes a lot of computing resources. Therefore, we proposed improved reptile meta-learning method. First, a multilingual unified word embedding method is proposed to represent multilingual knowledge. Secondly, the direction of meta-gradient is guided by calculating cumulative gradients on multiple specific tasks. In addition, the midpoint is taken as the meta-parameter in the space of the initial meta-parameter and the final task-specific model parameter to ensure that the meta-model has better multi-feature generalization ability. We conducted experiments in the CCMT2019 Mongolian-Chinese (Mo-Zh), Uyghur-Chinese (Uy-Zh) and Tibetan-Chinese (Ti-Zh), and the results show that our method has significantly improved the translation quality compared with the traditional methods.
What problem does this paper attempt to address?