Improving Neural Machine Translation with Linear Interpolation of a Short-Path Unit

LiYachao,LiJunhui,ZhangMin,LiYixin,ZouPeng

2020-01-01

Abstract:In neural machine translation (NMT), the source and target words are at the two ends of a large deep neural network, normally mediated by a series of non-linear activations. The problem with such c...

What problem does this paper attempt to address?

Improving Neural Machine Translation with Linear Interpolation of a Short-Path Unit.

Yachao Li,Junhui Li,Min Zhang,Yixin Li,Peng Zou

DOI: https://doi.org/10.1145/3377851

IF: 1.471

2020-01-01

ACM Transactions on Asian and Low-Resource Language Information Processing

Abstract:In neural machine translation (NMT), the source and target words are at the two ends of a large deep neural network, normally mediated by a series of non-linear activations. The problem with such consequent non-linear activations is that they significantly decrease the magnitude of the gradient in a deep neural network, and thus gradually loosen the interaction between source words and their translations. As a result, a source word may be incorrectly translated into a target word out of its translational equivalents. In this article, we propose short-path units (SPUs) to strengthen the association of source and target words by allowing information flow over adjacent layers effectively via linear interpolation. In particular, we enrich three critical NMT components with SPUs: (1) an enriched encoding model with SPU, which interpolates source word embeddings linearly into source annotations; (2) an enriched decoding model with SPU, which enables the source context linearly flow to target-side hidden states; and (3) an enriched output model with SPU, which further allows linear interpolation of target-side hidden states into output states. Experimentation on Chinese-to-English, English-to-German, and low-resource Tibetan-to-Chinese translation tasks demonstrates that the linear interpolation of SPUs significantly improves the overall translation quality by 1.88, 1.43, and 3.75 BLEU, respectively. Moreover, detailed analysis shows that our approaches much strengthen the association of source and target words. From the preceding, we can see that our proposed model is effective both in rich- and low-resource scenarios.
Deep Neural Machine Translation with Linear Associative Unit

Mingxuan Wang,Zhengdong Lu,Jie Zhou,Qun Liu

DOI: https://doi.org/10.48550/arXiv.1705.00861

2017-05-02

Computation and Language

Abstract:Deep Neural Networks (DNNs) have provably enhanced the state-of-the-art Neural Machine Translation (NMT) with their capability in modeling complex functions and capturing complex linguistic structures. However NMT systems with deep architecture in their encoder or decoder RNNs often suffer from severe gradient diffusion due to the non-linear recurrent activations, which often make the optimization much more difficult. To address this problem we propose novel linear associative units (LAU) to reduce the gradient propagation length inside the recurrent unit. Different from conventional approaches (LSTM unit and GRU), LAUs utilizes linear associative connections between input and output of the recurrent unit, which allows unimpeded information flow through both space and time direction. The model is quite simple, but it is surprisingly effective. Our empirical study on Chinese-English translation shows that our model with proper configuration can improve by 11.7 BLEU upon Groundhog and the best reported results in the same setting. On WMT14 English-German task and a larger WMT14 English-French task, our model achieves comparable results with the state-of-the-art.
Deep Learning-Based English-Chinese Translation Research

Yao Huang,Yi Xin

DOI: https://doi.org/10.1155/2022/3208167

IF: 2.223

2022-07-15

Advances in Meteorology

Abstract:Neural machine translation (NMT) has been bringing exciting news in the field of machine translation since its emergence. However, because NMT only employs single neural networks to convert natural languages, it suffers from two drawbacks in terms of reducing translation time: NMT is more sensitive to sentence length than statistical machine translation and the end-to-end implementation process fails to make explicit use of linguistic knowledge to improve translation performance. The network model performance of various deep learning machine translation tasks was constructed and compared in English-Chinese bilingual direction, and the defects of each network were solved by using an attention mechanism. The problems of gradient disappearance and gradient explosion are easy to occur in the recurrent neural network in the long-distance sequence. The short and long-term memory networks cannot reflect the information weight problems in long-distance sequences. In this study, through the comparison of examples, it is concluded that the introduction of an attention mechanism can improve the attention of context information in the process of model generation of the target language sequence, thus translating restore degree and fluency higher. This study proposes a neural machine translation method based on the divide-and-conquer strategy. Based on the idea of divide-and-conquer, this method identifies and extracts the longest noun phrase in a sentence and retains special identifiers or core words to form a sentence frame with the rest of the sentence. This method of translating the longest noun phrase and sentence frame separately by the neural machine translation system, and then recombining the translation, alleviates the poor performance of neural machine translation in long sentences. Experimental results show that the BLEU score of translation obtained by the proposed method has improved by 0.89 compared with the baseline method.

meteorology & atmospheric sciences
Improving Neural Machine Translation Model with Deep Encoding Information

Guiduo Duan,Haobo Yang,Ke Qin,Tianxi Huang

DOI: https://doi.org/10.1007/s12559-021-09860-7

IF: 4.89

2021-05-15

Cognitive Computation

Abstract:Availability of very high computational power along with the development of deep neural network (DNN) technology has enabled rapid progress of machine translation technology. The powerful representation ability of the deep neural network also enables the neural machine translation technology (NMT) to exploit the available large-scale bilingual parallel corpus as well as the computing power to provide a highly effective translation model. Nevertheless, the existing neural machine translation models only utilize the top layer encoder information, whereas the information available in deeper encoding layers is often ignored. This significantly constrains the performance of the translation model. To address this issue, in this paper, we propose a novel neural machine translation model which can fully exploit the deep encoding information. The core idea is to use different ways of aggregating the information from different encoding layers. We further design three different aggregation strategies including parallel layer, multi-layer, and dynamic layer encoding information aggregations. Three translation models are correspondingly trained and compared with the baseline transformer model for the Chinese-to-English translation task. The experimental results indicate that the BLEU-4 score of the proposed model has been increased by 0.89 compared with that of the benchmark model. Experiments demonstrate the effectiveness of the proposed method.

computer science, artificial intelligence,neurosciences
Improved Neural Machine Translation With Smt Features

Wei He,Zhongjun He,Hua Wu,Haifeng Wang

DOI: https://doi.org/10.1609/aaai.v30i1.9983

2016-01-01

Abstract:Neural machine translation (NMT) conducts end-to-end translation with a source language encoder and a target language decoder, making promising translation performance. However, as a newly emerged approach, the method has some limitations. An NMT system usually has to apply a vocabulary of certain size to avoid the time-consuming training and decoding, thus it causes a serious out-of-vocabulary problem. Furthermore, the decoder lacks a mechanism to guarantee all the source words to be translated and usually favors short translations, resulting in fluent but inadequate translations. In order to solve the above problems, we incorporate statistical machine translation (SMT) features, such as a translation model and an n-gram language model, with the NMT model under the log-linear framework. Our experiments show that the proposed method significantly improves the translation quality of the state-of-the-art NMT system on Chinese-to-English translation tasks. Our method produces a gain of up to 2.33 BLEU score on NIST open test sets.
Improving Low-Resource Neural Machine Translation with Weight Sharing

Tao Feng,Miao Li,Xiaojun Liu,Yichao Cao

DOI: https://doi.org/10.1007/978-3-030-01716-3_6

2018-01-01

Abstract:Neural machine translation (NMT) has achieved great success under a great deal of bilingual corpora in the past few years. However, it is much less effective for low-resource language. In order to alleviate the problem, we present two approaches which can improve the performance of low-resource NMT system. The first approach employs the weight sharing of decoder to enhance the target language model of low-resource NMT system. The second approach applies cross-lingual embedding and source sentence representation space sharing to strengthen the encoder of low-resource NMT. Our experiments demonstrate that the proposed method can obtain significant improvements on low-resource neural machine translation than baseline system. On the IWSLT2015 Vietnamese-English translation task, our model can improve the translation quality by an average of 1.43 BLEU scores. Besides, we can also get the increase of 0.96 BLEU scores when translating from Mongolian to Chinese.
Improving Neural Machine Translation with Neural Syntactic Distance

Chunpeng Ma,Akihiro Tamura,Masao Utiyama,Tiejun Zhao,Eiichiro Sumita

DOI: https://doi.org/10.18653/v1/n19-1205

2019-01-01

Abstract:The explicit use of syntactic information has been proved useful for neural machine translation (NMT). However, previous methods resort to either tree-structured neural networks or long linearized sequences, both of which are inefficient. Neural syntactic distance (NSD) enables us to represent a constituent tree using a sequence whose length is identical to the number of words in the sentence. NSD has been used for constituent parsing, but not in machine translation. We propose five strategies to improve NMT with NSD. Experiments show that it is not trivial to improve NMT with NSD; however, the proposed strategies are shown to improve translation performance of the baseline model (+2.1 (En-Ja), +1.3 (Ja-En), +1.2 (En-Ch), and +1.0 (Ch-En) BLEU).
Character-Aware Low-Resource Neural Machine Translation With Weight Sharing And Pre-Training

Yichao Cao,Miao Li,Tao Feng,Rujing Wang

DOI: https://doi.org/10.1007/978-3-030-32381-3_26

2019-01-01

Abstract:Neural Machine Translation (NMT) has recently achieved the state-of-the-art in many machine translation tasks, but one of the challenges that NMT faces is the lack of parallel corpora, especially for low-resource language pairs. And the result is that the performance of NMT is much less effective for low-resource languages. To address this specific problem, in this paper, we describe a novel NMT model that is based on encoder-decoder architecture and relies on character-level inputs. Our proposed model employs Convolutional Neural Networks (CNN) and highway networks over character inputs, whose outputs are given to an encoder-decoder neural machine translation network. Besides, we also present two other approaches to improve the performance of the low-resource NMT system much further. First, we use language modeling implemented by denoising autoencoding to pre-train and initialize the full model. Second, we share the weights of the front few layers of two encoders between two languages to strengthen the encoding ability of the model. We demonstrate our model on two low-resource language pairs. On the IWSLT2015 English-Vietnamese translation task, our proposed model obtains improvements up to 2.5 BLEU points compared to the baseline. We also outperform the baseline approach more than 3 BLEU points on the CWMT2018 Chinese-Mongolian translation task.
Language-aware Interlingua for Multilingual Neural Machine Translation.

Changfeng Zhu,Heng Yu,Shanbo Cheng,Weihua Luo

DOI: https://doi.org/10.18653/v1/2020.acl-main.150

2020-01-01

Abstract:Multilingual neural machine translation (NMT) has led to impressive accuracy improvements in low-resource scenarios by sharing common linguistic information across languages. However, the traditional multilingual model fails to capture the diversity and specificity of different languages, resulting in inferior performance compared with individual models that are sufficiently trained. In this paper, we incorporate a language-aware interlingua into the Encoder-Decoder architecture. The interlingual network enables the model to learn a language-independent representation from the semantic spaces of different languages, while still allowing for language-specific specialization of a particular language-pair. Experiments show that our proposed method achieves remarkable improvements over state-of-the-art multilingual NMT baselines and produces comparable performance with strong individual models.
Optimization of English Machine Translation by Deep Neural Network under Artificial Intelligence

Xiaohua Guo

DOI: https://doi.org/10.1155/2022/2003411

IF: 3.12

2022-04-21

Computational Intelligence and Neuroscience

Abstract:To improve the function of machine translation to adapt to global language translation, the work takes deep neural network (DNN) as the basic theory, carries out transfer learning and neural network translation modeling, and optimizes the word alignment function in machine translation performance. First, the work implements a deep learning translation network model for English translation. On this basis, the neural machine translation model is designed under transfer learning. The random shielding method is introduced to implement the language training model, and the machine translation is slightly adjusted as the goal of transfer learning, thereby improving the semantic understanding ability in translation performance. Meanwhile, the work design introduces the method of word alignment optimization and optimizes the performance of word alignment in the transformer system by using word corpus. The experimental results show that the proposed method reduces the average alignment error rate by 8.1%, 24.4%, and 22.1% in EnRo (English-Roman), EnGe (English-German), and EnFr (English-French), respectively, compared with the previous algorithms. Compared with the designed optimization method, the word alignment error rate is lower than that of traditional methods. The modeling and optimization method is feasible, which can effectively solve the problems of insufficient information utilization, large parameter scale, and difficult storage in the process of machine translation. Additionally, it provides a feasible idea and direction for the optimization and improvement in neural machine translation (NMT) system.

mathematical & computational biology,neurosciences
Towards Faster K-Nearest-neighbor Machine Translation

Xiangyu Shi,Yunlong Liang,Jinan Xu,Yufeng Chen

DOI: https://doi.org/10.54364/aaiml.2024.41111

2024-01-01

Advances in Artificial Intelligence and Machine Learning

Abstract:Recent works have proven the effectiveness of k-nearest-neighbor machine translation(a.k.akNN-MT) approaches to produce remarkable improvement in cross-domain translations.However, these models suffer from heavy retrieve overhead on the entire datastore whendecoding each token. We observe that during the decoding phase, about 67% to 84% oftokens are unvaried after searching over the corpus datastore, which means most of the tokenscause futile retrievals and introduce unnecessary computational costs by initiating k-nearest-neighbor searches. We consider this phenomenon is explainable in linguistics and propose asimple yet effective multi-layer perceptron (MLP) network to predict whether a token shouldbe translated jointly by the neural machine translation model and probabilities produced bythe kNN or just by the neural model. The results show that our method succeeds in reducingredundant retrieval operations and significantly reduces the overhead of kNN retrievals byup to 53% at the expense of a slight decline in translation quality. Moreover, our methodcould work together with all existing kNN-MT systems.
Improving Multilingual Translation by Representation and Gradient Regularization

Yilin Yang,Akiko Eriguchi,Alexandre Muzio,Prasad Tadepalli,Stefan Lee,Hany Hassan

DOI: https://doi.org/10.48550/arXiv.2109.04778

2022-01-19

Abstract:Multilingual Neural Machine Translation (NMT) enables one model to serve all translation directions, including ones that are unseen during training, i.e. zero-shot translation. Despite being theoretically attractive, current models often produce low quality translations -- commonly failing to even produce outputs in the right target language. In this work, we observe that off-target translation is dominant even in strong multilingual systems, trained on massive multilingual corpora. To address this issue, we propose a joint approach to regularize NMT models at both representation-level and gradient-level. At the representation level, we leverage an auxiliary target language prediction task to regularize decoder outputs to retain information about the target language. At the gradient level, we leverage a small amount of direct data (in thousands of sentence pairs) to regularize model gradients. Our results demonstrate that our approach is highly effective in both reducing off-target translation occurrences and improving zero-shot translation performance by +5.59 and +10.38 BLEU on WMT and OPUS datasets respectively. Moreover, experiments show that our method also works well when the small amount of direct data is not available.

Computation and Language,Artificial Intelligence
Automatic translation of spoken English based on improved machine learning algorithm

Lin Lin,Jie Liu,Xuebing Zhang,Xiufang Liang

DOI: https://doi.org/10.3233/jifs-189234

2021-02-02

Abstract:Due to the complexity of English machine translation technology and its broad application prospects, many experts and scholars have invested more energy to analyze it. In view of the complex and changeable English forms, the large difference between Chinese and English word order, and insufficient Chinese-English parallel corpus resources, this paper uses deep learning to complete the conversion between Chinese and English. The research focus of this paper is how to use language pairs with rich parallel corpus resources to improve the performance of Chinese-English neural machine translation, that is, to use multi-task learning to train neural machine translation models. Moreover, this research proposes a low-resource neural machine translation method based on weight sharing, which uses the weight-sharing method to improve the performance of Chinese-English low-resource neural machine translation. In addition, this study designs a control experiment to analyze the effectiveness of this study model. The research results show that the model proposed in this paper has a certain effect.
Enhanced Neural Machine Translation by Joint Decoding with Word and POS-tagging Sequences.

Feng Xiaocheng,Feng Zhangyin,Zhao Wanlong,Qin Bing,Liu Ting

DOI: https://doi.org/10.1007/s11036-020-01582-8

2020-01-01

Mobile Networks and Applications

Abstract:Machine translation has become an irreplaceable application in the use of mobile phones. However, the current mainstream neural machine translation models depend on continuously increasing the amount of parameters to achieve better performance, which is not applicable to the mobile phone. In this paper, we improve the performance of neural machine translation (NMT) with shallow syntax (e.g., POS tag) of target language, which has better accuracy and latency than deep syntax such as dependency parsing. In particular, our models take less parameters and runtime than other complex machine translation models, making mobile applications possible. In detail, we present three RNN-based NMT decoding models (independent decoder, gates shared decoder and fully shared decoder) to jointly predict target word and POS tag sequences. Experiments on Chinese-English and German-English translation tasks show that the fully shared decoder can acquire the best performance, which increases the BLEU score by 1.4 and 2.25 points respectively compared with the attention-based NMT model. In addition, we extend the idea to transformer-based models, and the experimental results also show that the BLEU score is further improved.
Neural Machine Translation Advised by Statistical Machine Translation

Xing Wang,Zhengdong Lu,Zhaopeng Tu,Hang Li,Deyi Xiong,Min Zhang

DOI: https://doi.org/10.48550/arXiv.1610.05150

2016-12-30

Abstract:Neural Machine Translation (NMT) is a new approach to machine translation that has made great progress in recent years. However, recent studies show that NMT generally produces fluent but inadequate translations (Tu et al. 2016b; Tu et al. 2016a; He et al. 2016; Tu et al. 2017). This is in contrast to conventional Statistical Machine Translation (SMT), which usually yields adequate but non-fluent translations. It is natural, therefore, to leverage the advantages of both models for better translations, and in this work we propose to incorporate SMT model into NMT framework. More specifically, at each decoding step, SMT offers additional recommendations of generated words based on the decoding information from NMT (e.g., the generated partial translation and attention history). Then we employ an auxiliary classifier to score the SMT recommendations and a gating function to combine the SMT recommendations with NMT generations, both of which are jointly trained within the NMT architecture in an end-to-end manner. Experimental results on Chinese-English translation show that the proposed approach achieves significant and consistent improvements over state-of-the-art NMT and SMT systems on multiple NIST test sets.

Computation and Language
Integrating Prior Translation Knowledge Into Neural Machine Translation

Kehai Chen,Rui Wang,Masao Utiyama,Eiichiro Sumita

DOI: https://doi.org/10.1109/TASLP.2021.3138714

2022-01-01

Abstract:Neural machine translation (NMT), which is an encoder-decoder joint neural language model with an attention mechanism, has achieved impressive results on various machine translation tasks in the past several years. However, the language model attribute of NMT tends to produce fluent yet sometimes unfaithful translations, which hinders the improvement of translation capacity. In response to this problem, we propose a simple and efficient method to integrate prior translation knowledge into NMT in a universal manner that is compatible with neural networks. Meanwhile, it enables NMT to consider the crossing language translation knowledge from the source-side of the training pipeline of NMT, thereby making full use of the prior translation knowledge to enhance the performance of NMT. The experimental results on two large-scale benchmark translation tasks demonstrated that our approach achieved a significant improvement over a strong baseline.
Improving Neural Sentence Alignment with Word Translation.

Ding Ying,Li Junhui,Gong Zhengxian,Zhou Guodong

DOI: https://doi.org/10.1007/s11704-019-9164-3

IF: 2.6688

2020-01-01

Frontiers of Computer Science

Abstract:Sentence alignment is a basic task in natural language processing which aims to extract high-quality parallel sentences automatically. Motivated by the observation that aligned sentence pairs contain a larger number of aligned words than unaligned ones, we treat word translation as one of the most useful external knowledge. In this paper, we show how to explicitly integrate word translation into neural sentence alignment. Specifically, this paper proposes three cross-lingual encoders to incorporate word translation: 1) Mixed Encoder that learns words and their translation annotation vectors over sequences where words and their translations are mixed alternatively; 2) Factored Encoder that views word translations as features and encodes words and their translations by concatenating their embeddings; and 3) Gated Encoder that uses gate mechanism to selectively control the amount of word translations moving forward. Experimentation on NIST MT and Opensubtitles Chinese-English datasets on both non-monotonicity and monotonicity scenarios demonstrates that all the proposed encoders significantly improve sentence alignment performance.
Multilingual Mix: Example Interpolation Improves Multilingual Neural Machine Translation

Yong Cheng,Ankur Bapna,Orhan Firat,Yuan Cao,Pidong Wang,Wolfgang Macherey

DOI: https://doi.org/10.48550/arXiv.2203.07627

2022-03-15

Abstract:Multilingual neural machine translation models are trained to maximize the likelihood of a mix of examples drawn from multiple language pairs. The dominant inductive bias applied to these models is a shared vocabulary and a shared set of parameters across languages; the inputs and labels corresponding to examples drawn from different language pairs might still reside in distinct sub-spaces. In this paper, we introduce multilingual crossover encoder-decoder (mXEncDec) to fuse language pairs at an instance level. Our approach interpolates instances from different language pairs into joint `crossover examples' in order to encourage sharing input and output spaces across languages. To ensure better fusion of examples in multilingual settings, we propose several techniques to improve example interpolation across dissimilar languages under heavy data imbalance. Experiments on a large-scale WMT multilingual dataset demonstrate that our approach significantly improves quality on English-to-Many, Many-to-English and zero-shot translation tasks (from +0.5 BLEU up to +5.5 BLEU points). Results on code-switching sets demonstrate the capability of our approach to improve model generalization to out-of-distribution multilingual examples. We also conduct qualitative and quantitative representation comparisons to analyze the advantages of our approach at the representation level.

Computation and Language,Artificial Intelligence
Improving Neural Machine Translation with Neural Sentence Rewriting

Tian Wu,Zhongjun He,Enhong Chen,Haifeng Wang

DOI: https://doi.org/10.1109/ialp.2018.8629254

2018-01-01

Abstract:A complex expression is more difficult for machine translation than a simplified sentence. One efficient method to handle this problem is to rewrite the source text into a simplified version before translation while keeping its meaning. In this paper, we propose a novel method to automatically rewrite source sentences based on neural machine translation. We propose a round-trip machine translation method to automatically generate a large amount of high quality rewritten pairs from bilingual corpus and then build an end-to-end sentence rewriting system based on neural network. Experimental results on Chinese-English IWSLT translation tasks show that our method leads to significant improvements over a strong baseline system.
Neural Machine Translation Research Based on the Semantic Vector of the Tri-Lingual Parallel Corpus

Xiaoxue Wang,Conghui Zhu,Sheng Li,Tiejun Zhao,Dequan Zheng

DOI: https://doi.org/10.1109/icmlc.2016.7860879

2016-01-01

Abstract:RNN Encoder-Decoder and attentional mechanism have lately been used to improve neural machine translation (NMT) on bilingual parallel corpus. In this paper, we propose tri-lingual NMT. Based on the Encoder-Decoder and attentional mechanism, we translate source language to target language, meanwhile translate another parallel source language to target language. We provides two approaches called splicing-model and similarity-model. Both of the approaches are in order to enhance the semantic representation of input sequences. Our experiments on the IWSLT 2012 Chinese to Japanese translation and English to Japanese tasks show that both of the methods provide a comparable or substantial improvement over the bilingual parallel corpus.

Improving Neural Machine Translation with Linear Interpolation of a Short-Path Unit

Improving Neural Machine Translation with Linear Interpolation of a Short-Path Unit.

Deep Neural Machine Translation with Linear Associative Unit

Deep Learning-Based English-Chinese Translation Research

Improving Neural Machine Translation Model with Deep Encoding Information

Improved Neural Machine Translation With Smt Features

Improving Low-Resource Neural Machine Translation with Weight Sharing

Improving Neural Machine Translation with Neural Syntactic Distance

Character-Aware Low-Resource Neural Machine Translation With Weight Sharing And Pre-Training

Language-aware Interlingua for Multilingual Neural Machine Translation.

Optimization of English Machine Translation by Deep Neural Network under Artificial Intelligence

Towards Faster K-Nearest-neighbor Machine Translation

Improving Multilingual Translation by Representation and Gradient Regularization

Automatic translation of spoken English based on improved machine learning algorithm

Enhanced Neural Machine Translation by Joint Decoding with Word and POS-tagging Sequences.

Neural Machine Translation Advised by Statistical Machine Translation

Integrating Prior Translation Knowledge Into Neural Machine Translation

Improving Neural Sentence Alignment with Word Translation.

Multilingual Mix: Example Interpolation Improves Multilingual Neural Machine Translation

Improving Neural Machine Translation with Neural Sentence Rewriting

Neural Machine Translation Research Based on the Semantic Vector of the Tri-Lingual Parallel Corpus