Abstract:Text-to-image (T2I) models can be maliciously used to generate harmful content such as sexually explicit, unfaithful, and misleading or Not-Safe-for-Work (NSFW) images. Previous attacks largely depend on the availability of the diffusion model or involve a lengthy optimization process. In this work, we investigate a more practical and universal attack that does not require the presence of a target model and demonstrate that the high-dimensional text embedding space inherently contains NSFW concepts that can be exploited to generate harmful images. We present the Jailbreaking Prompt Attack (JPA). JPA first searches for the target malicious concepts in the text embedding space using a group of antonyms generated by ChatGPT. Subsequently, a prefix prompt is optimized in the discrete vocabulary space to align malicious concepts semantically in the text embedding space. We further introduce a soft assignment with gradient masking technique that allows us to perform gradient ascent in the discrete vocabulary space. We perform extensive experiments with open-sourced T2I models, e.g. stable-diffusion-v1-4 and closed-sourced online services, e.g. DALLE2, Midjourney with black-box safety checkers. Results show that (1) JPA bypasses both text and image safety checkers (2) while preserving high semantic alignment with the target prompt. (3) JPA demonstrates a much faster speed than previous methods and can be executed in a fully automated manner. These merits render it a valuable tool for robustness evaluation in future text-to-image generation research.

What problem does this paper attempt to address?

The paper aims to address the safety issues of text-to-image (T2I) diffusion models in generating harmful content such as pornography, violence, or other inappropriate content for work environments. Specifically, the paper proposes a method called "Jailbreaking Prompt Attack" (JPA), which can implement more practical and general attacks without the need for the target model to be present. Unlike previous attack methods that rely on the availability of diffusion models or require long optimization processes, JPA achieves its goals through the following steps: 1. **Concept Embedding Search**: First, search for the target malicious concept in the text embedding space, using a set of antonyms generated by ChatGPT to assist in this process. 2. **Prefix Prompt Optimization**: Then, optimize a prefix prompt in the discrete vocabulary space to semantically align the malicious concept in the text embedding space. 3. **Soft Assignment and Gradient Masking Technique**: Introduce a technique of soft assignment combined with gradient masking, allowing gradient ascent to be performed in the discrete vocabulary space. Through these steps, JPA can not only bypass existing text and image safety checkers but also maintain a high degree of semantic consistency with the target prompt, demonstrating faster speed and fully automated execution capabilities compared to previous methods. Additionally, experimental results show that JPA performs excellently on multiple open-source and closed-source T2I models, including online services like Stable Diffusion and DALL·E 2. In summary, this study reveals that existing safety measures are less effective than expected and proposes a new attack method to evaluate the robustness of future T2I generation models.

Jailbreaking Prompt Attack: A Controllable Adversarial Attack against Diffusion Models

Multimodal Pragmatic Jailbreak on Text-to-image Models

Perception-guided Jailbreak against Text-to-Image Models

SneakyPrompt: Jailbreaking Text-to-image Generative Models

Chain-of-Jailbreak Attack for Image Generation Models via Editing Step by Step

AdvI2I: Adversarial Image Attack on Image-to-Image Diffusion models

Divide-and-Conquer Attack: Harnessing the Power of LLM to Bypass the Censorship of Text-to-Image Generation Model

Antelope: Potent and Concealed Jailbreak Attack Strategy

IDEATOR: Jailbreaking Large Vision-Language Models Using Themselves

Automatic Jailbreaking of the Text-to-Image Generative AI Systems

SurrogatePrompt: Bypassing the Safety Filter of Text-to-Image Models via Substitution

IDEATOR: Jailbreaking VLMs Using VLMs

Deciphering the Chaos: Enhancing Jailbreak Attacks via Adversarial Prompt Translation

On the Proactive Generation of Unsafe Images From Text-To-Image Models Using Benign Prompts

Jailbreak Vision Language Models via Bi-Modal Adversarial Prompt

Divide-and-Conquer Attack: Harnessing the Power of LLM to Bypass Safety Filters of Text-to-Image Models

Revealing Vulnerabilities in Stable Diffusion via Targeted Attacks

Jailbreaking Text-to-Image Models with LLM-Based Agents

Jailbreak Large Vision-Language Models Through Multi-Modal Linkage

Text-to-Image Diffusion Models can be Easily Backdoored through Multimodal Data Poisoning

BSPA: Exploring Black-box Stealthy Prompt Attacks Against Image Generators