Abstract:Deep transformer neural network models have improved the predictive accuracy of intelligent text processing systems in the biomedical domain. They have obtained state-of-the-art performance scores on a wide variety of biomedical and clinical Natural Language Processing (NLP) benchmarks. However, the robustness and reliability of these models has been less explored so far. Neural NLP models can be easily fooled by adversarial samples, i.e. minor changes to input that preserve the meaning and understandability of the text but force the NLP system to make erroneous decisions. This raises serious concerns about the security and trust-worthiness of biomedical NLP systems, especially when they are intended to be deployed in real-world use cases. We investigated the robustness of several transformer neural language models, i.e. BioBERT, SciBERT, BioMed-RoBERTa, and Bio-ClinicalBERT, on a wide range of biomedical and clinical text processing tasks. We implemented various adversarial attack methods to test the NLP systems in different attack scenarios. Experimental results showed that the biomedical NLP models are sensitive to adversarial samples; their performance dropped in average by 21 and 18.9 absolute percent on character-level and word-level adversarial noise, respectively, on Micro-F1, Pearson Correlation, and Accuracy measures. Conducting extensive adversarial training experiments, we fine-tuned the NLP models on a mixture of clean samples and adversarial inputs. Results showed that adversarial training is an effective defense mechanism against adversarial noise; the models’ robustness improved in average by 11.3 absolute percent. In addition, the models’ performance on clean data increased in average by 2.4 absolute percent, demonstrating that adversarial training can boost generalization abilities of biomedical NLP systems. This study takes an important step towards revealing vulnerabilities of deep neural language models in biomedical NLP applications. It also provides practical and effective strategies to develop secure, trust-worthy, and accurate intelligent text processing systems in the biomedical domain.

Power in Numbers: Robust reading comprehension by finetuning with four adversarial sentences per example

Adversarial Examples for Evaluating Reading Comprehension Systems

A Robust Adversarial Training Approach to Machine Reading Comprehension

Misleading Sentiment Analysis: Generating Adversarial Texts by the Ensemble Word Addition Algorithm

The Impacts of Unanswerable Questions on the Robustness of Machine Reading Comprehension Models

Adversarial Augmentation Policy Search for Domain and Cross-Lingual Generalization in Reading Comprehension

Commonsense knowledge adversarial dataset that challenges ELECTRA

Improving Machine Reading Comprehension via Adversarial Training

Improving the Robustness of Deep Reading Comprehension Models by Leveraging Syntax Prior.

SenTest: Evaluating Robustness of Sentence Encoders

Using Adversarial Attacks to Reveal the Statistical Bias in Machine Reading Comprehension Models

Adversarial Training for Improving Model Robustness? Look at Both Prediction and Interpretation

Robust Reading Comprehension with Linguistic Constraints Via Posterior Regularization.

R&R: Metric-guided Adversarial Sentence Generation

Adversarial Examples with Difficult Common Words for Paraphrase Identification

Adversarial Training for Commonsense Inference

Towards Improving Adversarial Training of NLP Models

Benchmarking Robustness of Machine Reading Comprehension Models

Combining permuted language model and adversarial training for Chinese machine reading comprehension

It Is Not About What You Say, It Is About How You Say It: A Surprisingly Simple Approach for Improving Reading Comprehension

Improving the robustness and accuracy of biomedical language models through adversarial training