Safeguard Text-to-Image Diffusion Models with Human Feedback Inversion

Sanghyun Kim,Seohyeon Jung,Balhae Kim,Moonseok Choi,Jinwoo Shin,Juho Lee

2024-07-17

Abstract:This paper addresses the societal concerns arising from large-scale text-to-image diffusion models for generating potentially harmful or copyrighted content. Existing models rely heavily on internet-crawled data, wherein problematic concepts persist due to incomplete filtration processes. While previous approaches somewhat alleviate the issue, they often rely on text-specified concepts, introducing challenges in accurately capturing nuanced concepts and aligning model knowledge with human understandings. In response, we propose a framework named Human Feedback Inversion (HFI), where human feedback on model-generated images is condensed into textual tokens guiding the mitigation or removal of problematic images. The proposed framework can be built upon existing techniques for the same purpose, enhancing their alignment with human judgment. By doing so, we simplify the training objective with a self-distillation-based technique, providing a strong baseline for concept removal. Our experimental results demonstrate our framework significantly reduces objectionable content generation while preserving image quality, contributing to the ethical deployment of AI in the public sphere.

Computer Vision and Pattern Recognition,Artificial Intelligence

What problem does this paper attempt to address?

The problem that this paper attempts to solve is the social concerns raised by large - scale text - to - image diffusion models when generating potentially harmful or copyright - protected content. Existing models rely heavily on data crawled from the Internet, and there may be problems with incomplete filtering of concepts in this data. Although previous methods have alleviated this problem to a certain extent, they usually rely on text - specified concepts, which leads to challenges in accurately capturing subtle concepts and keeping the model's knowledge in line with human understanding.

To solve these problems, the authors propose a framework named Human Feedback Inversion (HFI), which guides the reduction or removal of problematic images by converting human feedback on model - generated images into text tokens. This framework can be built on existing technologies and enhance its consistency with human judgment. By simplifying the training objective and adopting self - distillation technology, HFI provides a powerful concept - removal baseline. Experimental results show that this framework significantly reduces the generation of offensive content while maintaining image quality, which is helpful for the ethical deployment of AI in the public domain.

### Formula Display

- **Forward Process of Diffusion Model**:
\[
q(x_t|x_{t - 1})=\mathcal{N}(x_t;\sqrt{1-\beta_t}x_{t - 1},\beta_tI)
\]
\[
q(x_T|x_0)\approx\mathcal{N}(x_T;0,I)
\]

- **Reverse Process of Diffusion Model**:
\[
p_\theta(x_{0:T}) = p(x_T)\prod_{t = 1}^T p_\theta(x_{t - 1}|x_t)
\]

- **Loss Function of Diffusion Model**:
\[
L_{\text{DM}}(\theta)=\mathbb{E}_{x_0,\epsilon,t}\left[\left\|\epsilon-\epsilon_\theta(\sqrt{\bar{\alpha}_t}x_0+\sqrt{1-\bar{\alpha}_t}\epsilon;t)\right\|^2_2\right]
\]
where \(\epsilon\sim\mathcal{N}(0,I)\), \(\bar{\alpha}_t=\prod_{s = 1}^t(1-\beta_s)\), \(t\sim U([1,T])\)

- **Mean Squared Error Loss of Reward Model**:
\[
L_{\text{MSE}}(\psi)=\mathbb{E}_{(x,y)\sim D_{\text{human}}}\left[(r_\psi(x)-y)^2\right]
\]

- **Negative Log - Likelihood Loss of Reward Model**:
\[
L_{\text{NLL}}(\psi)=-\frac{1}{\binom{M}{2}}\mathbb{E}_{(x^+,x^-)\sim D_{\text{human}}}\log\sigma(r_\psi(x^+)-r_\psi(x^-))
\]

- **Feedback Embedding Optimization Objective**:
\[
v^*=\arg\min_v\mathbb{E}_{x\sim p_\theta(x|c)}\left[-r_\psi(x)\log p_\theta(x|v)\right]
\]

- **Loss Function of Self - Distillation Diffusion Model**:
\[
L_{\text{SDD}}=\mathbb{E}_{x_t\sim p_{\theta^*}(x_t|v^*),t}\left[\left\|\epsilon\right.\right]