EigenSR: Eigenimage-Bridged Pre-Trained RGB Learners for Single Hyperspectral Image Super-Resolution

Xi Su,Xiangfei Shen,Mingyang Wan,Jing Nie,Lihui Chen,Haijun Liu,Xichuan Zhou

2024-09-06

Abstract:Single hyperspectral image super-resolution (single-HSI-SR) aims to improve the resolution of a single input low-resolution HSI. Due to the bottleneck of data scarcity, the development of single-HSI-SR lags far behind that of RGB natural images. In recent years, research on RGB SR has shown that models pre-trained on large-scale benchmark datasets can greatly improve performance on unseen data, which may stand as a remedy for HSI. But how can we transfer the pre-trained RGB model to HSI, to overcome the data-scarcity bottleneck? Because of the significant difference in the channels between the pre-trained RGB model and the HSI, the model cannot focus on the correlation along the spectral dimension, thus limiting its ability to utilize on HSI. Inspired by the HSI spatial-spectral decoupling, we propose a new framework that first fine-tunes the pre-trained model with the spatial components (known as eigenimages), and then infers on unseen HSI using an iterative spectral regularization (ISR) to maintain the spectral correlation. The advantages of our method lie in: 1) we effectively inject the spatial texture processing capabilities of the pre-trained RGB model into HSI while keeping spectral fidelity, 2) learning in the spectral-decorrelated domain can improve the generalizability to spectral-agnostic data, and 3) our inference in the eigenimage domain naturally exploits the spectral low-rank property of HSI, thereby reducing the complexity. This work bridges the gap between pre-trained RGB models and HSI via eigenimages, addressing the issue of limited HSI training data, hence the name EigenSR. Extensive experiments show that EigenSR outperforms the state-of-the-art (SOTA) methods in both spatial and spectral metrics. Our code will be released.

Image and Video Processing,Computer Vision and Pattern Recognition

What problem does this paper attempt to address?

The paper attempts to address the performance bottleneck issue caused by the scarcity of training data in the single low-resolution hyperspectral image (HSI) super-resolution (single-HSI-SR) task. Specifically:

1. **Data Scarcity Issue**: Compared to RGB natural images, the dataset size of hyperspectral images is smaller, which limits the development of single HSI super-resolution techniques.
2. **RGB Model Transfer Challenge**: Existing research shows that pre-trained RGB models can significantly improve performance when handling unseen data. However, due to the significant difference in the number of channels between HSI and RGB images, directly transferring RGB models to HSI is challenging.
3. **Spectral Correlation Preservation**: Since HSI has multiple spectral channels, a method is needed to maintain the correlation between these spectral channels.

To address these issues, the authors propose the EigenSR framework, which utilizes Singular Value Decomposition (SVD) to decompose HSI into spatial parts (i.e., feature images) and spectral parts. This allows the use of large-scale pre-trained RGB models in the feature image domain and maintains spectral consistency during inference through Iterative Spectral Regularization (ISR). This approach not only effectively enhances the handling of spatial details but also preserves the integrity of spectral information. Experimental results show that EigenSR outperforms existing state-of-the-art methods in both spatial and spectral metrics.