Breaking ReAct Agents: Foot-in-the-Door Attack Will Get You In

Itay Nakash,George Kour,Guy Uziel,Ateret Anaby-Tavor

2024-10-22

Abstract:Following the advancement of large language models (LLMs), the development of LLM-based autonomous agents has become increasingly prevalent. As a result, the need to understand the security vulnerabilities of these agents has become a critical task. We examine how ReAct agents can be exploited using a straightforward yet effective method we refer to as the foot-in-the-door attack. Our experiments show that indirect prompt injection attacks, prompted by harmless and unrelated requests (such as basic calculations) can significantly increase the likelihood of the agent performing subsequent malicious actions. Our results show that once a ReAct agents thought includes a specific tool or action, the likelihood of executing this tool in the subsequent steps increases significantly, as the agent seldom re-evaluates its actions. Consequently, even random, harmless requests can establish a foot-in-the-door, allowing an attacker to embed malicious instructions into the agents thought process, making it more susceptible to harmful directives. To mitigate this vulnerability, we propose implementing a simple reflection mechanism that prompts the agent to reassess the safety of its actions during execution, which can help reduce the success of such attacks.

Cryptography and Security,Artificial Intelligence

What problem does this paper attempt to address?

The problem that this paper attempts to solve is the security vulnerabilities in large - language models (LLMs) and the autonomous agents built on these models. Specifically, the authors have studied how to manipulate ReAct agents by using a method called "Foot - in - the - Door (FITD)" attack. This attack, through Indirect Prompt Injection (IPI), that is, through harmless and seemingly unrelated requests (such as basic calculations), significantly increases the probability of the agent performing subsequent malicious actions. The paper points out that once a specific tool or action is included in the ReAct agent's thought, the probability of performing these tools or actions in subsequent steps will increase substantially, because the agent rarely re - evaluates its actions. Therefore, even random, harmless requests can provide an "opportunity for the attacker to get a foot in the door", enabling the attacker to embed malicious instructions into the agent's thought process, making it more vulnerable to harmful instructions. To mitigate this vulnerability, the authors propose implementing a simple reflection mechanism to prompt the agent to re - evaluate the security of its actions during the execution process, so as to help reduce the success rate of such attacks.