Test-time Training for Hyperspectral Image Super-resolution

Ke Li,Luc Van Gool,Dengxin Dai

2024-09-13

Abstract:The progress on Hyperspectral image (HSI) super-resolution (SR) is still lagging behind the research of RGB image SR. HSIs usually have a high number of spectral bands, so accurately modeling spectral band interaction for HSI SR is hard. Also, training data for HSI SR is hard to obtain so the dataset is usually rather small. In this work, we propose a new test-time training method to tackle this problem. Specifically, a novel self-training framework is developed, where more accurate pseudo-labels and more accurate LR-HR relationships are generated so that the model can be further trained with them to improve performance. In order to better support our test-time training method, we also propose a new network architecture to learn HSI SR without modeling spectral band interaction and propose a new data augmentation method Spectral Mixup to increase the diversity of the training data at test time. We also collect a new HSI dataset with a diverse set of images of interesting objects ranging from food to vegetation, to materials, and to general scenes. Extensive experiments on multiple datasets show that our method can improve the performance of pre-trained models significantly after test-time training and outperform competing methods significantly for HSI SR.

Computer Vision and Pattern Recognition

What problem does this paper attempt to address?

The problems that this paper attempts to solve are: **The progress of hyperspectral image (HSI) super - resolution (SR) technology lags behind the research on RGB image super - resolution, especially the challenges in dealing with the interactions between spectral bands and obtaining large - scale training data**.

Specifically, hyperspectral images usually have a large number of spectral bands, so it is very difficult to accurately model the interactions between spectral bands. In addition, due to the high cost and complex equipment for obtaining high - resolution hyperspectral images, the scale of datasets available for training is usually small. These problems make it difficult for existing hyperspectral image super - resolution models to generalize to new data samples during testing.

To solve these problems, the author proposes a **test - time training method**, aiming to improve the effect of hyperspectral image super - resolution in the following ways:
1. **Self - training framework**: Generate more accurate pseudo - labels and more accurate low - resolution - high - resolution relationships, thereby further training the model to improve performance.
2. **New network architecture**: A simple grouped super - resolution network focusing on spatial super - resolution is designed, which avoids modeling the interactions between spectral bands, thereby improving the stability of training.
3. **Spectral Mixup data augmentation method**: Create new spectral images by mixing the contents of different spectral bands, increasing the diversity of training data without destroying the detailed structure of hyperspectral images.

These methods work together to significantly improve the performance of the pre - trained model during testing and outperform other existing methods on multiple benchmark datasets.