Abstract:Recent studies have demonstrated outstanding capabilities of large language models (LLMs) in software engineering domain, covering numerous tasks such as code generation and comprehension. While the benefit of LLMs for coding task is well noted, it is perceived that LLMs are vulnerable to adversarial attacks. In this paper, we study the specific LLM vulnerability to imperceptible character attacks, a type of prompt-injection attack that uses special characters to befuddle an LLM whilst keeping the attack hidden to human eyes. We devise four categories of attacks and investigate their effects on the performance outcomes of tasks relating to code analysis and code comprehension. Two generations of ChatGPT are included to evaluate the impact of advancements made to contemporary models. Our experimental design consisted of comparing perturbed and unperturbed code snippets and evaluating two performance outcomes, which are model confidence using log probabilities of response, and correctness of response. We conclude that earlier version of ChatGPT exhibits a strong negative linear correlation between the amount of perturbation and the performance outcomes, while the recent ChatGPT presents a strong negative correlation between the presence of perturbation and performance outcomes, but no valid correlational relationship between perturbation budget and performance outcomes. We anticipate this work contributes to an in-depth understanding of leveraging LLMs for coding tasks. It is suggested future research should delve into how to create LLMs that can return a correct response even if the prompt exhibits perturbations.

Robustness Evaluation of Cloud-Deployed Large Language Models Against Chinese Adversarial Text Attacks

Robustness of Large Language Models Against Adversarial Attacks

Adversarial Attacks on Large Language Model-Based System and Mitigating Strategies: A Case Study on ChatGPT

Improving Adversarial Robustness of 3D Point Cloud Classification Models

Towards Robustness of Large Language Models on Text-to-SQL Task: an Adversarial and Cross-Domain Investigation

KGPA: Robustness Evaluation for Large Language Models via Cross-Domain Knowledge Graphs

On the Robustness of ChatGPT: An Adversarial and Out-of-distribution Perspective

On Evaluating Adversarial Robustness of Large Vision-Language Models

Assessing Adversarial Robustness of Large Language Models: An Empirical Study

Adversarial Attacks and Defenses in Large Language Models: Old and New Threats

Towards Evaluating the Robustness of Chinese BERT Classifiers

Adversarial GLUE: A Multi-Task Benchmark for Robustness Evaluation of Language Models

On Adversarial Robustness of Language Models in Transfer Learning

Who Can Withstand Chat-Audio Attacks? An Evaluation Benchmark for Large Language Models

RCC: A Paradigm for Training a Robust Chinese Text Classification Model

How Robust is Google's Bard to Adversarial Image Attacks?

Offline Textual Adversarial Attacks Against Large Language Models

Robustness Over Time: Understanding Adversarial Examples' Effectiveness on Longitudinal Versions of Large Language Models

What You See Is Not Always What You Get: An Empirical Study of Code Comprehension by Large Language Models

Defending Large Language Models Against Attacks With Residual Stream Activation Analysis

GPSAttack: A Unified Glyphs, Phonetics and Semantics Multi-Modal Attack against Chinese Text Classification Models