Abstract:Text classification systems have been proven vulnerable to adversarial text examples, modified versions of the original text examples that are often unnoticed by human eyes, yet can force text classification models to alter their classification. Often, research works quantifying the impact of adversarial text attacks have been applied only to models trained in English. In this paper, we introduce the first word-level study of adversarial attacks in Arabic. Specifically, we use a synonym (word-level) attack using a Masked Language Modeling (MLM) task with a BERT model in a black-box setting to assess the robustness of the state-of-the-art text classification models to adversarial attacks in Arabic. To evaluate the grammatical and semantic similarities of the newly produced adversarial examples using our synonym BERT-based attack, we invite four human evaluators to assess and compare the produced adversarial examples with their original examples. We also study the transferability of these newly produced Arabic adversarial examples to various models and investigate the effectiveness of defense mechanisms against these adversarial examples on the BERT models. We find that fine-tuned BERT models were more susceptible to our synonym attacks than the other Deep Neural Networks (DNN) models like WordCNN and WordLSTM we trained. We also find that fine-tuned BERT models were more susceptible to transferred attacks. We, lastly, find that fine-tuned BERT models successfully regain at least 2% in accuracy after applying adversarial training as an initial defense mechanism.

Detecting textual adversarial examples through text modification on text classification systems

Misleading Sentiment Analysis: Generating Adversarial Texts by the Ensemble Word Addition Algorithm

Repairing Adversarial Texts Through Perturbation

On Adversarial Examples for Text Classification by Perturbing Latent Representations

WordChange: Adversarial Examples Generation Approach for Chinese Text Classification

Detection of Word Adversarial Examples in Text Classification: Benchmark and Baseline via Robust Density Estimation

Detecting Textual Adversarial Examples Based on Distributional Characteristics of Data Representations

Evaluating Text Classification Robustness to Part-of-Speech Adversarial Examples

A Modified Word Saliency-Based Adversarial Attack on Text Classification Models

Identifying Adversarial Attacks on Text Classifiers

Generating Natural Language Adversarial Examples Through Probability Weighted Word Saliency

WordRevert: Adversarial Examples Defence Method for Chinese Text Classification

On the (Statistical) Detection of Adversarial Examples

Rule-based adversarial sample generation for text classification

Generating Black-Box Adversarial Examples for Text Classifiers Using a Deep Reinforced Model

TextTricker: Loss-based and gradient-based adversarial attacks on text classification models

Towards Crafting Text Adversarial Samples

Reversible Jump Attack to Textual Classifiers with Modification Reduction

MPAT: Building Robust Deep Neural Networks against Textual Adversarial Attacks

Arabic Synonym BERT-based Adversarial Examples for Text Classification