Abstract:Subject-driven text-to-image generation has witnessed remarkable advancements in its ability to learn and capture characteristics of a subject using only a limited number of images. However, existing methods commonly rely on high-quality images for training and may struggle to generate reasonable images when the input images are blemished by artifacts. This is primarily attributed to the inadequate capability of current techniques in distinguishing subject-related features from disruptive artifacts. In this paper, we introduce ArtiFade to tackle this issue and successfully generate high-quality artifact-free images from blemished datasets. Specifically, ArtiFade exploits fine-tuning of a pre-trained text-to-image model, aiming to remove artifacts. The elimination of artifacts is achieved by utilizing a specialized dataset that encompasses both unblemished images and their corresponding blemished counterparts during fine-tuning. ArtiFade also ensures the preservation of the original generative capabilities inherent within the diffusion model, thereby enhancing the overall performance of subject-driven methods in generating high-quality and artifact-free images. We further devise evaluation benchmarks tailored for this task. Through extensive qualitative and quantitative experiments, we demonstrate the generalizability of ArtiFade in effective artifact removal under both in-distribution and out-of-distribution scenarios.

What problem does this paper attempt to address?

### What problem does this paper attempt to solve? This paper aims to solve the problem that existing methods are difficult to generate high - quality images in subject - driven text - to - image generation when the input images are blemished. Specifically: 1. **Limitations of existing methods**: - Current methods usually rely on high - quality images for training. For images with blemishes (such as visible blemishes like watermarks, graffiti, stickers, and invisible blemishes like adversarial noise), these methods may not be able to generate reasonable images. - These methods lack the ability to distinguish between subject - related features and interfering blemishes, resulting in a decline in the quality of the generated images. 2. **New problems proposed**: - The paper defines a new problem: how to use blemished images for blemished subject - driven generation. This problem is very important in practical applications because obtaining blemish - free images is often very expensive or even impossible. ### Solutions To solve the above problems, the paper proposes a model named **ArtiFade**. ArtiFade mainly solves the problems in the following ways: 1. **Dataset construction**: - A special dataset is constructed, which contains blemish - free images and their corresponding blemished versions. These paired images are used to fine - tune various subject - driven generation methods. 2. **Model fine - tuning**: - ArtiFade fine - tunes the pre - trained text - to - image generation models (such as diffusion models) to align the implicit relationships between blemish - free images and blemished images. - An artifact - free embedding is introduced to enhance the fidelity of the prompts. 3. **Evaluation benchmark**: - A set of evaluation benchmarks is proposed, including multiple test sets with different blemishes and customized evaluation metrics, to accurately evaluate the performance of blemished subject - driven generation methods. 4. **Experimental verification**: - Through extensive qualitative and quantitative experiments, the effectiveness and generalization ability of ArtiFade in dealing with blemished images are proved, both in in - distribution and out - of - distribution scenarios. ### Summary The main contribution of the paper is that it solves the blemished subject - driven generation problem for the first time and proposes the ArtiFade model, which can effectively extract subject - specific information from blemished training data to generate high - quality and blemish - free images. In addition, the paper also introduces an evaluation benchmark for this task and verifies the superior performance of ArtiFade through experiments.

ArtiFade: Learning to Generate High-quality Subject from Blemished Images

DiffGAR: Model-Agnostic Restoration from Generative Artifacts Using Image-to-Image Diffusion Models

Fine Tuning Text-to-Image Diffusion Models for Correcting Anomalous Images

Diffusion Brush: A Latent Diffusion Model-based Editing Tool for AI-generated Images

EARN: Toward Efficient and Robust JPEG Compression Artifact Reduction

Joint Generative Image Deblurring Aided by Edge Attention Prior and Dynamic Kernel Selection

Learning to Restore Multiple Image Degradations Simultaneously

Beyond Pixels: Text Enhances Generalization in Real-World Image Restoration

Detecting Human Artifacts from Text-to-Image Models

ReDiFine: Reusable Diffusion Finetuning for Mitigating Degradation in the Chain of Diffusion

Artifact Restoration in Histology Images with Diffusion Probabilistic Models

Playground v2.5: Three Insights towards Enhancing Aesthetic Quality in Text-to-Image Generation

AuthFace: Towards Authentic Blind Face Restoration with Face-oriented Generative Diffusion Prior

FT-Shield: A Watermark Against Unauthorized Fine-tuning in Text-to-Image Diffusion Models

HiFi Tuner: High-Fidelity Subject-Driven Fine-Tuning for Diffusion Models

LatentArtiFusion: An Effective and Efficient Histological Artifacts Restoration Framework

Efficient Degradation-aware Any Image Restoration

DifFace: Blind Face Restoration with Diffused Error Contraction

DreamArtist: Towards Controllable One-Shot Text-to-Image Generation via Positive-Negative Prompt-Tuning

Beyond Imperfections: A Conditional Inpainting Approach for End-to-End Artifact Removal in VTON and Pose Transfer

Referring Flexible Image Restoration