Abstract:To solve the low spatial and/or temporal resolution problem which the conventional hyperspectral cameras often suffer from, spectral compressive imaging systems (SCI) have attracted more attention recently. Recovering a hyperspectral image (HSI) from its corresponding 2D coded image is an ill-posed inverse problem, and learning accurate prior from HSI and 2D coded image is essential to solve this inverse problem. Existing methods only use supervised networks that focus on learning generalized prior from training datasets, or only use unsupervised networks that focus on learning specific prior from 2D coded image, resulting in the inability to learn both generalized and specific priors. Also, when learning the priors, existing methods cannot simultaneously give consideration to both global and local scales, as well as both spatial and spectral dimensions. To cope with this problem, in this paper, we propose a Supervised-Unsupervised Combined Transformer Network (SUCTNet) composed by a supervised Spatio-spectral Transformer network (SSTNet) and an Unsupervised Multi-level Feature Refinement network (UMFRNet). Specifically, we first develop the SSTNet to learn generalized prior and obtain a preliminary HSI. In SSTNet, the proposed spatial encoding and spectral decoding network architecture enables it to simultaneously consider both spatial and spectral dimensions, and a proposed Global and Local Multi head Self Attention block (GL-MSA) enables it simultaneously to consider both global and local scales. Then, the preliminary HSI is fed into the proposed UMFRNet to learn specific prior and obtain the target HSI. In UMFRNet, a proposed multi-level feature refinement mechanism and the physical imaging model of SCI are used to improve reconstruction accuracy and generalization performance. Extensive experiments show that our method significantly outperforms state-of-the-art (SOTA) methods on simulated and real datasets. Codes will be available at https://github.com/Vzhouhan/SUCTNet .

SSTU: Swin-Spectral Transformer U-Net for Hyperspectral Whole Slide Image Reconstruction

CSwT-SR: Conv-Swin Transformer for Blind Remote Sensing Image Super-Resolution with Amplitude-Phase Learning and Structural Detail Alternating Learning

SWUNet: Swin Transformer Based UNet for Hyperspectral Reconstruction

Dual Self-Attention Swin Transformer for Hyperspectral Image Super-Resolution

Spectral Swin Transformer Network for Hyperspectral Image Classification

A Swin Transformer-Based Fusion Approach for Hyperspectral Image Super-Resolution

Supervised-unsupervised combined transformer for spectral compressive imaging reconstruction

SpecTr: Spectral Transformer for Microscopic Hyperspectral Pathology Image Segmentation

Spectral-Enhanced Sparse Transformer Network for Hyperspectral Super-Resolution Reconstruction

FCSwinU: Fourier Convolutions and Swin Transformer UNet for Hyperspectral and Multispectral Image Fusion

A spectral and spatial transformer for hyperspectral remote sensing image super-resolution

Spectral-Swin Transformer with Spatial Feature Extraction Enhancement for Hyperspectral Image Classification

HST: Hierarchical Swin Transformer for Compressed Image Super-resolution

Coarse-to-Fine Sparse Transformer for Hyperspectral Image Reconstruction

Pixel Adaptive Deep Unfolding Transformer for Hyperspectral Image Reconstruction

Hyperspectral and multispectral images fusion based on pyramid swin transformer

A novel spatial and spectral transformer network for hyperspectral image super-resolution

Spatial–Spectral 1dswin Transformer with Groupwise Feature Tokenization for Hyperspectral Image Classification

SSTrans-Net: Smart Swin Transformer Network for medical image segmentation

Pixel-Based Long-Wave Infrared Spectral Image Reconstruction Using a Hierarchical Spectral Transformer

Dynamic Spectral Guided Spatial Sparse Transformer for Hyperspectral Image Reconstruction