Abstract:Pre-trained language model is one of the most important models in the natural language processing field, as pre-train-finetune has become the paradigm in various NLP downstream tasks.Previous studies have proved integrating pre-trained language models (e.g., BERT) into neural machine translation (NMT) models can improve translation performance.However, it is still unclear whether these improvements stem from enhanced semantic or syntactic modeling capabilities, as well as how pre-trained knowledge impacts the robustness of the models.To address these questions, a systematic study was conducted to examine the syntactic ability of BERT-enhanced NMT models using probing tasks.The study revealed that the enhanced models showed proficiency in modeling word order, highlighting their syntactic modeling capabilities.In addition, an attacking method was proposed to evaluate the robustness of NMT models in handling word order.BERT-enhanced NMT models yielded better translation performance in most of the tasks, indicating that BERT can improve the robustness of NMT models.It was observed that BERT-enhanced NMT model generated poorer translations than vanilla NMT model after attacking in the English-German translation task, which meant that English BERT worsened model robustness in such a scenario.Further analyses revealed that English BERT failed to bridge the semantic gap between the original and perturbed sources, leading to more copying errors and errors in translating low-frequency words.These findings suggest that the benefits of pre-training may not always be consistent in downstream tasks, and careful consideration should be given to its usage.

How Robust Are Character-Based Word Embeddings in Tagging and MT Against Wrod Scramlbing or Randdm Nouse?

Methods for Estimating and Improving Robustness of Language Models

Addressing the Vulnerability of NMT in Input Perturbations

Research on the Robustness of Neural Machine Translation Systems in Word Order Perturbation

Did Translation Models Get More Robust Without Anyone Even Noticing?

Robust Neural Machine Translation: Modeling Orthographic and Interpunctual Variation

Robustness-Eva-MRC: Assessing and Analyzing the Robustness of Neural Models in Extractive Machine Reading Comprehension

Robustness of LLMs to Perturbations in Text

Enhancing Model Robustness Via Lexical Distilling

Word Shape Matters: Robust Machine Translation with Visual Embedding

Robust Neural Machine Translation for Clean and Noisy Speech Transcripts

Domain Robustness in Neural Machine Translation

Towards Robust Neural Machine Translation

SenTest: Evaluating Robustness of Sentence Encoders

Certified Robustness to Adversarial Word Substitutions

Is Robustness Transferable across Languages in Multilingual Neural Machine Translation?

Whispers of Doubt Amidst Echoes of Triumph in NLP Robustness

Robust Textual Embedding Against Word-level Adversarial Attacks

Robust Natural Language Processing: Recent Advances, Challenges, and Future Directions

Stumbling Blocks: Stress Testing the Robustness of Machine-Generated Text Detectors Under Attacks

Training on Synthetic Noise Improves Robustness to Natural Noise in Machine Translation