Abstract:Pre-trained models with large-scale training data, such as CLIP and Stable Diffusion, have demonstrated remarkable performance in various high-level computer vision tasks such as image understanding and generation from language descriptions. Yet, their potential for low-level tasks such as image restoration remains relatively unexplored. In this paper, we explore such models to enhance image restoration. As off-the-shelf features (OSF) from pre-trained models do not directly serve image restoration, we propose to learn an additional lightweight module called Pre-Train-Guided Refinement Module (PTG-RM) to refine restoration results of a target restoration network with OSF. PTG-RM consists of two components, Pre-Train-Guided Spatial-Varying Enhancement (PTG-SVE), and Pre-Train-Guided Channel-Spatial Attention (PTG-CSA). PTG-SVE enables optimal short- and long-range neural operations, while PTG-CSA enhances spatial-channel attention for restoration-related learning. Extensive experiments demonstrate that PTG-RM, with its compact size ($<$1M parameters), effectively enhances restoration performance of various models across different tasks, including low-light enhancement, deraining, deblurring, and denoising.

What problem does this paper attempt to address?

The problem that this paper attempts to solve is to utilize the prior knowledge in pre - trained models to enhance the restoration effect in image restoration tasks. Specifically, the paper points out that although large - scale pre - trained models such as CLIP and Stable Diffusion perform excellently in high - level computer vision tasks, their potential in low - level tasks such as image restoration has not been fully explored yet. For this reason, the author proposes a new method. By introducing a lightweight module - the Pre - Train - Guided Refinement Module (PTG - RM), the features of these pre - trained models are utilized to improve the effect of image restoration. ### Main Problems 1. **Challenges in Image Restoration**: - Image restoration is an ill - posed problem, and it is difficult to significantly improve performance only by modifying the network structure or increasing model parameters. - Existing image restoration methods rely on strong priors, such as noise level or blur kernel, but these priors are difficult to estimate in practical scenarios and are not robust. 2. **Utilizing the Priors of Pre - trained Models**: - Pre - trained models are trained on large - scale data and may have been exposed to various degraded images, so they may contain useful restoration - related information. - How to effectively utilize the features of these pre - trained models to enhance the image restoration effect is a key issue. ### Solutions 1. **Pre - Train - Guided Refinement Module (PTG - RM)**: - PTG - RM consists of two components: Pre - Train - Guided Spatial Variation Enhancement (PTG - SVE) and Pre - Train - Guided Channel - Spatial Attention (PTG - CSA). - PTG - SVE optimizes the short - range and long - range neural operation ranges through spatial variation operations, thereby more effectively fusing the features of different regions. - PTG - CSA further enhances the restoration results through channel and spatial attention mechanisms, so that the features of different regions are appropriately focused on. 2. **Technical Details**: - **PTG - SVE**: Use the features $ g $ of the pre - trained model to predict the optimal neural operation range at each position, extract short - range and long - range features through convolution and transformer operations respectively, and fuse these features according to the predicted range score map $ M $. - **PTG - CSA**: Utilize the features $ g $ of the pre - trained model to generate channel and spatial attention maps and further optimize the feature representation. 3. **Experimental Verification**: - The author has carried out extensive experiments on multiple image restoration tasks, including low - light enhancement, rain removal, deblurring and denoising. - The experimental results show that PTG - RM can significantly improve the performance of various models on different tasks, and its number of parameters is very small (<1M). ### Main Contributions 1. Propose a general method to utilize the prior knowledge of pre - trained models to enhance various image restoration tasks. 2. Introduce a new paradigm to formulate effective neural operation ranges and attention mechanisms through pre - training priors. 3. Verify the effectiveness of the method through extensive experiments, showing significant improvements on different datasets, networks and tasks. In conclusion, by introducing PTG - RM, this paper successfully solves the problem of how to utilize the prior knowledge of pre - trained models to enhance the image restoration effect, providing a new and effective method for the field of image restoration.

Boosting Image Restoration via Priors from Pre-trained Models

Training-Free Large Model Priors for Multiple-in-One Image Restoration

Image Restoration using Feature-guidance

Re-boosting Self-Collaboration Parallel Prompt GAN for Unsupervised Image Restoration

Textual Prompt Guided Image Restoration

Parameter Efficient Adaptation for Image Restoration with Heterogeneous Mixture-of-Experts

A Dive into SAM Prior in Image Restoration

Priors in Deep Image Restoration and Enhancement: A Survey

Prompt-In-Prompt Learning for Universal Image Restoration

ProRes: Exploring Degradation-aware Visual Prompt for Universal Image Restoration

Improving Image Restoration through Removing Degradations in Textual Representations

Wide & deep learning for spatial & intensity adaptive image restoration

TAPE: Task-Agnostic Prior Embedding for Image Restoration

Learning from History: Task-agnostic Model Contrastive Learning for Image Restoration

Plug-and-Play Image Restoration with Deep Denoiser Prior

A Restoration Network as an Implicit Prior

Image Restoration Based on End-to-End Unrolled Network

Enhanced Image Restoration Via Supervised Target Feature Transfer

Multimodal Prompt Perceiver: Empower Adaptiveness, Generalizability and Fidelity for All-in-One Image Restoration

LoRA-IR: Taming Low-Rank Experts for Efficient All-in-One Image Restoration

Revitalizing Convolutional Network for Image Restoration