PromptHSI: Universal Hyperspectral Image Restoration Framework for Composite Degradation

Chia-Ming Lee,Ching-Heng Cheng,Yu-Fan Lin,Yi-Ching Cheng,Wo-Ting Liao,Chih-Chung Hsu,Fu-En Yang,Yu-Chiang Frank Wang

2024-11-25

Abstract:Recent developments in All-in-One (AiO) RGB image restoration and prompt learning have enabled the representation of distinct degradations through prompts, allowing degraded images to be effectively addressed by a single restoration model. However, this paradigm faces significant challenges when transferring to hyperspectral image (HSI) restoration tasks due to: 1) the domain gap between RGB and HSI features and difference on their structures, 2) information loss in visual prompts under severe composite degradations, and 3) difficulties in capturing HSI-specific degradation representations through text prompts. To address these challenges, we propose PromptHSI, the first universal AiO HSI restoration framework. By leveraging the frequency-aware feature modulation based on characteristics of HSI degradations, we decompose text prompts into intensity and bias controllers to effectively guide the restoration process while avoiding domain gaps. Our unified architecture excels at both fine-grained recovery and global information restoration tasks. Experimental results demonstrate superior performance under various degradation combinations, indicating great potential for practical remote sensing applications. The source code and dataset will be publicly released.

Image and Video Processing,Computer Vision and Pattern Recognition

What problem does this paper attempt to address?

The main problem that this paper attempts to solve is the poor performance of existing hyperspectral image (HSI) restoration methods in dealing with compound degradations. Specifically, the article points out:

1. **Domain gap between RGB and HSI features**: RGB image restoration models are difficult to be directly applied to HSI restoration because there are significant differences in structure and characteristics between the two.
2. **Information loss of visual cues under severe compound degradations**: Traditional visual cue methods have limited effectiveness in the face of multiple complex degradations, resulting in unsatisfactory restoration effects.
3. **Difficulty in capturing HSI - specific degradation representations through text cues**: Existing text cue methods cannot effectively capture the degradation patterns unique to HSI.

To solve these problems, the authors propose a general - purpose hyperspectral image restoration framework named PromptHSI. This framework has the following features:

- **Frequency - aware feature modulation**: Based on the frequency - domain characteristics of HSI degradation, the text cues are decomposed into intensity and bias controllers, thereby effectively guiding the restoration process and avoiding the domain gap.
- **Unified architecture**: It can handle both fine - grained restoration (such as spatial - spectral super - resolution) and global information restoration (such as cloud removal, inpainting) tasks simultaneously.
- **Strong generalization ability**: Through flexible text cue control, it shows strong adaptability to unseen degradation types.

In summary, PromptHSI aims to overcome the key challenges in HSI processing by combining prompt learning and visual - language model frameworks, achieve controllable restoration of compound degradations, and provide support for practical applications.

### Formula summary

Some important formulas involved in the article are as follows:

1. **Fourier transform**:
\[
F(I_{\text{clean}}) = F_s(I_{\text{clean}}) + iF_c(I_{\text{clean}})
\]
where \( F_s \) and \( F_c \) represent the sine and cosine components respectively.

2. **Affine transformation of degradation effects**:
\[
F(I_{\text{degraded}}) = \hat{\lambda} \odot F(I_{\text{clean}}) + \hat{\mu} = (1 + \lambda) \odot F(I_{\text{clean}}) + \mu
\]
where \( \odot \) represents element - wise multiplication, \( \lambda \) is the scaling factor, and \( \mu \) is the bias term.

3. **Frequency - domain feature modulation**:
\[
\hat{F_f} = (1 + \lambda_l^m) \odot F_f^l + (1 + \lambda_h^m) \odot F_f^h + \mu
\]
where \( F_f^l \) and \( F_f^h \) are the low - frequency and high - frequency components respectively.

4. **Total loss function**:
\[
\ell_{\text{total}} = \lambda_1 \ell_1 + \lambda_2 \ell_{\text{SAM}} + \lambda_3 \ell_{\text{SWT}} + \lambda_4 \ell_{\text{BMSE}}
\]

These formulas help to understand how PromptHSI improves the restoration effect through frequency - aware feature modulation and ensures its robustness and consistency under different degradation combinations.