Abstract:Fusing a low spatial resolution hyperspectral image (LR-HIS) with a high spatial resolution multispectral image has become popular for generating a high spatial resolution hyperspectral image (HR-HSI). Most methods assume that the degradation information from high resolution to low resolution is known in spatial and spectral domains. Conversely, this information is often limited or unavailable in practice, restricting their performance. Furthermore, existing fusion methods still face the problem of insufficient exploration of the cross-interaction between the spatial and spectral domains in the HR-HSI, leaving scope for further improvement. This article proposes an unsupervised hybrid network of transformer and CNN (uHNTC) for blind HSI-MSI fusion. The uHNTC comprises three subnetworks: a transformer-based feature fusion subnetwork (FeafusFomer) and two CNN-based degradation subnetworks (SpaDNet and SpeDNet). Considering the strong multilevel spatio-spectral correlation between the desired HR-HSI and the observed images, we design a multilevel cross-feature attention (MCA) mechanism in FeafusFormer. By incorporating the hierarchical spatio-spectral feature fusion into the attention mechanism in the transformer, the MCA globally keeps a high spatio-spectral cross-similarity between the recovered HR-HSI and observed images, thereby ensuring the high cross-interaction of the recovered HR-HSI. Subsequently, the characteristics of degradation information are utilized to guide the design of the SpaDNet and SpeDNet, which helps FeafusFormer accurately recover the desired HR-HSI in complex real-world environments. Through an unsupervised joint training of the three subnetworks, uHNTC recovers the desired HR-HSI without preknown degradation information. Experimental results on three public datasets and a WorldView-2 images show that the uHNTC outperforms ten state-of-the-art fusion methods. Code available: https://github.com/Caoxuheng/HIFtool.

Transformer-Driven Inverse Problem Transform for Fast Blind Hyperspectral Image Dehazing

Vision Transformers for Single Image Dehazing

An Efficient Dehazing Algorithm Based on the Fusion of Transformer and Convolutional Neural Network.

Hybrid Spectral Denoising Transformer with Guided Attention

Frequency-Oriented Transformer for Remote Sensing Image Dehazing

Visual transformer with stable prior and patch-level attention for single image dehazing

TransDehaze: transformer-enhanced texture attention for end-to-end single image dehaze

Hider: A Hyperspectral Image Denoising Transformer With Spatial-Spectral Constraints for Hybrid Noise Removal

EHA-Transformer: Efficient and Haze-Adaptive Transformer for Single Image Dehazing

Three-Dimension Spatial–Spectral Attention Transformer for Hyperspectral Image Denoising

Low-Rank Prompt-Guided Transformer for Hyperspectral Image Denoising

A Data-Centric Solution to NonHomogeneous Dehazing via Vision Transformer

Adaptive haze pixel intensity perception transformer structure for image dehazing networks

Multiscale-Sparse Spatial-Spectral Transformer for Hyperspectral Image Denoising

An Unsupervised Dehazing Network With Hybrid Prior Constraints for Hyperspectral Image

Satellite Image Dehazing Based on Dual Frequency Pass Networks

Pixel Adaptive Deep Unfolding Transformer for Hyperspectral Image Reconstruction

Unsupervised Hybrid Network of Transformer and CNN for Blind Hyperspectral and Multispectral Image Fusion

Hyperspectral Image Transformer Classification Networks

Pixel Transformer for Synthetic-to-Real Single Image Dehazing

S^2-Transformer for Mask-Aware Hyperspectral Image Reconstruction