Abstract:Hyperspectral imaging has the potential to improve intraoperative decision making if tissue characterisation is performed in real-time and with high-resolution. Hyperspectral snapshot mosaic sensors offer a promising approach due to their fast acquisition speed and compact size. However, a demosaicking algorithm is required to fully recover the spatial and spectral information of the snapshot images. Most state-of-the-art demosaicking algorithms require ground-truth training data with paired snapshot and high-resolution hyperspectral images, but such imagery pairs with the exact same scene are physically impossible to acquire in intraoperative settings. In this work, we present a fully unsupervised hyperspectral image demosaicking algorithm which only requires exemplar snapshot images for training purposes. We regard hyperspectral demosaicking as an ill-posed linear inverse problem which we solve using a deep neural network. We take advantage of the spectral correlation occurring in natural scenes to design a novel inter spectral band regularisation term based on spatial gradient consistency. By combining our proposed term with standard regularisation techniques and exploiting a standard data fidelity term, we obtain an unsupervised loss function for training deep neural networks, which allows us to achieve real-time hyperspectral image demosaicking. Quantitative results on hyperspetral image datasets show that our unsupervised demosaicking approach can achieve similar performance to its supervised counter-part, and significantly outperform linear demosaicking. A qualitative user study on real snapshot hyperspectral surgical images confirms the results from the quantitative analysis. Our results suggest that the proposed unsupervised algorithm can achieve promising hyperspectral demosaicking in real-time thus advancing the suitability of the modality for intraoperative use.

ADMM-DSP: A Deep Spectral Image Prior for Snapshot Spectral Image Demosaicing

Spectral Imaging with Deep Learning.

PPI Edge Infused Spatial-Spectral Adaptive Residual Network for Multispectral Filter Array Image Demosaicing

Learning spectral-spatial prior via 3ddncnn for hyperspectral image deconvolution

A Snapshot Multi-Spectral Demosaicing Method for Multi-Spectral Filter Array Images Based on Channel Attention Network

Prior Image Guided Snapshot Compressive Spectral Imaging

Unsupervised Spectral Demosaicing with Lightweight Spectral Attention Networks

Effective Snapshot Compressive-spectral Imaging via Deep Denoising and Total Variation Priors

Hyperspectral Image Denoising Via Double Subspace Deep Prior

A Snapshot Near-Infrared Hyperspectral Demosaicing Method with Convolutional Neural Networks in Low Illumination Environment

Fast Multispectral Imaging by Spatial Pixel-Binning and Spectral Unmixing

A Deep Joint Network for Multispectral Demosaicking Based on Pseudo-Panchromatic Images

NTIRE 2022 Spectral Demosaicing Challenge and Data Set

Hyperspectral Unmixing Powered by Deep Image Priors and Denoising Regularization.

Multispectral Demosaicing Via Non-Local Low-Rank Regularization

Combining Low-Rank and Deep Plug-and-Play Priors for Snapshot Compressive Imaging

Spatial gradient consistency for unsupervised learning of hyperspectral demosaicking: Application to surgical imaging

A Fast Alternating Minimization Algorithm for Coded Aperture Snapshot Spectral Imaging Based on Sparsity and Deep Image Priors

Generalized Inpainting Method for Hyperspectral Image Acquisition

Hyperspectral Image Restoration Via a New Symmetric ADMM Approach

Optimized High-Frequency Based Interpolation for Multispectral Demosaicking