One Prompt Word is Enough to Boost Adversarial Robustness for Pre-trained Vision-Language Models

Lin Li,Haoyan Guan,Jianing Qiu,Michael Spratling

2024-03-04

Abstract:Large pre-trained Vision-Language Models (VLMs) like CLIP, despite having remarkable generalization ability, are highly vulnerable to adversarial examples. This work studies the adversarial robustness of VLMs from the novel perspective of the text prompt instead of the extensively studied model weights (frozen in this work). We first show that the effectiveness of both adversarial attack and defense are sensitive to the used text prompt. Inspired by this, we propose a method to improve resilience to adversarial attacks by learning a robust text prompt for VLMs. The proposed method, named Adversarial Prompt Tuning (APT), is effective while being both computationally and data efficient. Extensive experiments are conducted across 15 datasets and 4 data sparsity schemes (from 1-shot to full training data settings) to show APT's superiority over hand-engineered prompts and other state-of-the-art adaption methods. APT demonstrated excellent abilities in terms of the in-distribution performance and the generalization under input distribution shift and across datasets. Surprisingly, by simply adding one learned word to the prompts, APT can significantly boost the accuracy and robustness (epsilon=4/255) over the hand-engineered prompts by +13% and +8.5% on average respectively. The improvement further increases, in our most effective setting, to +26.4% for accuracy and +16.7% for robustness. Code is available at

Computer Vision and Pattern Recognition,Artificial Intelligence,Machine Learning

What problem does this paper attempt to address?

The problem this paper attempts to address is: although large pre-trained Vision-Language Models (VLMs) like CLIP have excellent generalization capabilities, they are very vulnerable to adversarial examples. This paper studies the adversarial robustness of VLMs from the perspective of text prompts and proposes a method to improve VLMs' resistance to adversarial attacks by learning robust text prompts, namely Adversarial Prompt Tuning (APT).

Specifically, the paper focuses on the following aspects:
1. **Sensitivity to Adversarial Attacks and Defenses**: Research shows that the effectiveness of both adversarial attacks and defenses is highly sensitive to the text prompts used.
2. **Proposed Method**: The APT method is proposed, which improves the model's adversarial robustness by learning a robust text prompt. The APT method performs well in terms of computational and data efficiency.
3. **Experimental Validation**: Extensive experiments were conducted on 15 datasets and 4 data sparsity schemes (ranging from 1-shot to full training data settings) to validate the superiority of the APT method over hand-engineered prompts and other state-of-the-art adaptation methods.
4. **Performance Improvement**: By simply adding a learned word to the prompt, APT can significantly improve accuracy and robustness. In the most effective setting, accuracy increased by 26.4% and robustness by 16.7%.

In summary, this paper aims to enhance the adversarial robustness of pre-trained VLMs by improving text prompts, thereby increasing the reliability and security of the models in practical applications.