Improving Image Restoration through Removing Degradations in Textual Representations

Jingbo Lin,Zhilu Zhang,Yuxiang Wei,Dongwei Ren,Dongsheng Jiang,Wangmeng Zuo

DOI: https://doi.org/10.48550/arXiv.2312.17334

2023-12-28

Computer Vision and Pattern Recognition

Abstract:In this paper, we introduce a new perspective for improving image restoration by removing degradation in the textual representations of a given degraded image. Intuitively, restoration is much easier on text modality than image one. For example, it can be easily conducted by removing degradation-related words while keeping the content-aware words. Hence, we combine the advantages of images in detail description and ones of text in degradation removal to perform restoration. To address the cross-modal assistance, we propose to map the degraded images into textual representations for removing the degradations, and then convert the restored textual representations into a guidance image for assisting image restoration. In particular, We ingeniously embed an image-to-text mapper and text restoration module into CLIP-equipped text-to-image models to generate the guidance. Then, we adopt a simple coarse-to-fine approach to dynamically inject multi-scale information from guidance to image restoration networks. Extensive experiments are conducted on various image restoration tasks, including deblurring, dehazing, deraining, and denoising, and all-in-one image restoration. The results showcase that our method outperforms state-of-the-art ones across all these tasks. The codes and models are available at \url{https://github.com/mrluin/TextualDegRemoval}.

What problem does this paper attempt to address?

The paper attempts to address the problem of how to improve image restoration performance by removing degradation in text representations in image restoration tasks. Specifically, the authors propose a new perspective: first performing restoration in the text modality, as degradation is more loosely coupled with content in this modality, making it easier to remove degradation. Then, the restored content is used to guide the image restoration process. This approach aims to overcome challenges present in existing image restoration methods, such as conflicts when handling multiple degradations and the tight coupling between degradation and the desired image content. By doing so, the authors hope to improve the quality of image restoration, particularly in tasks like deblurring, dehazing, deraining, and denoising.

The main contributions of the paper include:
1. Introducing a new perspective on image restoration, where restoration is first performed in the text space, and the restored content is then used to guide image restoration.
2. Proposing the embedding of an image-to-text mapper and a text restoration module into a CLIP-equipped text-to-image model to generate clear guiding images from degraded images.
3. Extensive experiments on multiple tasks demonstrate that this method improves the performance of existing state-of-the-art image restoration networks.