Abstract:To prevent unauthorized use of text in images, Scene Text Removal (STR) has become a crucial task. It focuses on automatically removing text and replacing it with a natural, text-less background while preserving significant details such as texture, color, and contrast. Despite its importance in privacy protection, STR faces several challenges, including boundary artifacts, inconsistent texture and color, and preserving correct shadows. Most STR approaches estimate a text region mask to train a model, solving for image translation or inpainting to generate a text-free image. Thus, the quality of the generated image depends on the accuracy of the inpainting mask and the generator's capability. In this work, we leverage the superior capabilities of diffusion models in generating high-quality, consistent images to address the STR problem. We introduce a ControlNet diffusion model, treating STR as an inpainting task. To enhance the model's robustness, we develop a mask pretraining pipeline to condition our diffusion model. This involves training a masked autoencoder (MAE) using a combination of box masks and coarse stroke masks, and fine-tuning it using masks derived from our novel segmentation-based mask refinement framework. This framework iteratively refines an initial mask and segments it using the SLIC and Hierarchical Feature Selection (HFS) algorithms to produce an accurate final text mask. This improves mask prediction and utilizes rich textural information in natural scene images to provide accurate inpainting masks. Experiments on the SCUT-EnsText and SCUT-Syn datasets demonstrate that our method significantly outperforms existing state-of-the-art techniques.
What problem does this paper attempt to address?
The problem that this paper attempts to solve is the challenges in Scene Text Removal (STR). Specifically, the goal of STR is to automatically remove text in an image and replace it with a natural, text - free background while maintaining important details such as texture, color, and contrast. Although STR is very important in terms of privacy protection, it faces some challenges, such as boundary artifacts, inconsistent texture and color, and correct shadow preservation.
### Main Problems and Challenges of the Paper
1. **Boundary Artifacts**: When removing text from a scene, boundary residues may be left as artifacts.
2. **Inconsistent Texture and Color**: In the absence of observed boundary artifacts, the text - removed area may not obtain a texture and color consistent with the neighboring area.
3. **Shadow Preservation**: Ensure that the shadow is still correctly preserved in the image after text removal.
### Limitations of Existing Methods
Most existing STR methods train models by estimating text - area masks to solve image - translation or - inpainting problems. The quality of the generated image depends on the accuracy of the inpainting mask and the ability of the generator. However, these methods are insufficient in dealing with structural and textural consistency.
### Solutions
To solve the above problems, this paper introduces a diffusion - model - based method - DiffSTR, treating STR as an inpainting task. Specific improvements include:
- **Utilizing Diffusion Models**: Diffusion models perform excellently in generating high - quality, consistent images, so they are used to solve the STR problem.
- **Introducing ControlNet**: By introducing ControlNet to enhance the robustness of the diffusion model and improve the inpainting quality.
- **Pretraining Pipeline**: Develop a mask pretraining pipeline, train the Mask Auto - Encoder (MAE) using a combined box - mask and rough - stroke - mask, and fine - tune it through a novel segmentation - mask - refinement framework.
- **Mask - Refinement Framework**: Propose a segmentation - based mask - refinement framework, iteratively refine the initial mask and use the SLIC and Hierarchical Feature Selection (HFS) algorithms for segmentation to generate an accurate final text mask.
### Experimental Results
Experiments show that DiffSTR significantly outperforms the existing state - of - the - art methods on the SCUT - EnsText and SCUT - Syn datasets, especially achieving better results in metrics such as PSNR and MSSIM, proving its effectiveness in real - scene text - removal tasks.
### Formula Representation
To ensure the correctness and readability of formulas, the following are some formula examples involved in the paper:
- Objective Function:
\[
\mathcal{L}=\mathbb{E}_{(x_0),(x_i),y_{\text{MRF}},x_h,t,\epsilon_h}\left[\left\|\epsilon-\epsilon_\theta(z,z_i,y_{\text{MRF}},x_h,t)\right\|_2^2\right]
\]
- Noise - Adding Process:
\[
z_t = z+\text{Noise}(t)
\]
- Loss Function:
\[
\mathcal{L}_{\text{MAE}}=\mathbb{E}_{(x_{\text{gt}},x_c)}\left[\left\|x_{\text{gt}}-x_c\right\|_2^2\right]
\]
Through these improvements, DiffSTR successfully solves the key challenges in STR and shows superior performance on multiple datasets.