Abstract:Vision Language Models (VLMs) have revolutionized the creation of generalist web agents, empowering them to autonomously complete diverse tasks on real-world websites, thereby boosting human efficiency and productivity. However, despite their remarkable capabilities, the safety and security of these agents against malicious attacks remain critically underexplored, raising significant concerns about their safe deployment. To uncover and exploit such vulnerabilities in web agents, we provide AdvWeb, a novel black-box attack framework designed against web agents. AdvWeb trains an adversarial prompter model that generates and injects adversarial prompts into web pages, misleading web agents into executing targeted adversarial actions such as inappropriate stock purchases or incorrect bank transactions, actions that could lead to severe real-world consequences. With only black-box access to the web agent, we train and optimize the adversarial prompter model using DPO, leveraging both successful and failed attack strings against the target agent. Unlike prior approaches, our adversarial string injection maintains stealth and control: (1) the appearance of the website remains unchanged before and after the attack, making it nearly impossible for users to detect tampering, and (2) attackers can modify specific substrings within the generated adversarial string to seamlessly change the attack objective (e.g., purchasing stocks from a different company), enhancing attack flexibility and efficiency. We conduct extensive evaluations, demonstrating that AdvWeb achieves high success rates in attacking SOTA GPT-4V-based VLM agent across various web tasks. Our findings expose critical vulnerabilities in current LLM/VLM-based agents, emphasizing the urgent need for developing more reliable web agents and effective defenses. Our code and data are available at <a class="link-external link-https" href="https://ai-secure.github.io/AdvWeb/" rel="external noopener nofollow">this https URL</a> .

Poison Attacks and Adversarial Prompts Against an Informed University Virtual Assistant

Adversarial Attacks on Large Language Model-Based System and Mitigating Strategies: A Case Study on ChatGPT

Imposter.AI: Adversarial Attacks with Hidden Intentions towards Aligned Large Language Models

RatGPT: Turning online LLMs into Proxies for Malware Attacks

An Early Categorization of Prompt Injection Attacks on Large Language Models

Poisoning Programs by Un-Repairing Code: Security Concerns of AI-generated Code

Forcing Generative Models to Degenerate Ones: The Power of Data Poisoning Attacks

Toxicity in ChatGPT: Analyzing Persona-assigned Language Models

Evil Geniuses: Delving into the Safety of LLM-based Agents

Bot-adversarial dialogue for safe conversational agents

AgentPoison: Red-teaming LLM Agents via Poisoning Memory or Knowledge Bases

PoisonedRAG: Knowledge Corruption Attacks to Retrieval-Augmented Generation of Large Language Models

Concealed Data Poisoning Attacks on NLP Models

Not what you've signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection

Robust Testing of AI Language Model Resiliency with Novel Adversarial Prompts

Robust Conversational Agents against Imperceptible Toxicity Triggers

Vulnerabilities in AI Code Generators: Exploring Targeted Data Poisoning Attacks

AdvWeb: Controllable Black-box Attacks on VLM-powered Web Agents

SoK: Prompt Hacking of Large Language Models

Query-Based Adversarial Prompt Generation

Imprompter: Tricking LLM Agents into Improper Tool Use