Abstract:The technology of hyperspectral imaging (HSI) records the visual information upon long-range-distributed spectral wavelengths. A representative hyperspectral image acquisition procedure conducts a 3D-to-2D encoding by the coded aperture snapshot spectral imager (CASSI) and requires a software decoder for the 3D signal reconstruction. By observing this physical encoding procedure, two major challenges stand in the way of a high-fidelity reconstruction. (i) To obtain 2D measurements, CASSI dislocates multiple channels by disperser-titling and squeezes them onto the same spatial region, yielding an entangled data loss. (ii) The physical coded aperture leads to a masked data loss by selectively blocking the pixel-wise light exposure. To tackle these challenges, we propose a spatial-spectral (S^2-) Transformer network with a mask-aware learning strategy. First, we simultaneously leverage spatial and spectral attention modeling to disentangle the blended information in the 2D measurement along both two dimensions. A series of Transformer structures are systematically designed to fully investigate the spatial and spectral informative properties of the hyperspectral data. Second, the masked pixels will induce higher prediction difficulty and should be treated differently from unmasked ones. Thereby, we adaptively prioritize the loss penalty attributing to the mask structure by inferring the pixel-wise reconstruction difficulty upon the mask-encoded prediction. We theoretically discusses the distinct convergence tendencies between masked/unmasked regions of the proposed learning strategy. Extensive experiments demonstrates that the proposed method achieves superior reconstruction performance. Additionally, we empirically elaborate the behaviour of spatial and spectral attentions under the proposed architecture, and comprehensively examine the impact of the mask-aware learning.

Improving Spectral Snapshot Reconstruction with Spectral-Spatial Rectification

Snapshot Spectral Imaging Based on Aberration Model-Driven Deep Learning

A hyperspectral image denoising method based on land cover spectral autocorrelation

A spectral and spatial transformer for hyperspectral remote sensing image super-resolution

Mask-guided Spectral-wise Transformer for Efficient Hyperspectral Image Reconstruction

Spectral Reconstruction Network From Multispectral Images to Hyperspectral Images: A Multitemporal Case

SPECAT: SPatial-spEctral Cumulative-Attention Transformer for High-Resolution Hyperspectral Image Reconstruction

Spectral Reconstruction From Satellite Multispectral Imagery Using Convolution and Transformer Joint Network

Spectral Superresolution Using Transformer with Convolutional Spectral Self-Attention

Hyperspectral Compressive Snapshot Reconstruction via Coupled Low-Rank Subspace Representation and Self-Supervised Deep Network

Remote Sensing Hyperspectral Image Super-Resolution via Multidomain Spatial Information and Multiscale Spectral Information Fusion

Spectral Query Spatial: Revisiting the Role of Center Pixel in Transformer for Hyperspectral Image Classification

Dynamic Spectral Guided Spatial Sparse Transformer for Hyperspectral Image Reconstruction

Coarse-to-Fine Sparse Transformer for Hyperspectral Image Reconstruction

Tensor-Based Sparse Representation for Hyperspectral Image Reconstruction Using RGB Inputs

S^2-Transformer for Mask-Aware Hyperspectral Image Reconstruction

MST++: Multi-stage Spectral-wise Transformer for Efficient Spectral Reconstruction

A Recurrent Feedback Hyperspectral Image Super-Resolution Reconstruction Method by Using Self-Attention-Based Pixel Awareness

Hyperspectral Image Superresolution via Deep Structure and Texture Interfusion

Spectral-Enhanced Sparse Transformer Network for Hyperspectral Super-Resolution Reconstruction

Hyperspectral Image Denoising via Spatial–Spectral Recurrent Transformer