Abstract:Hyperspectral imaging has the potential to improve intraoperative decision making if tissue characterisation is performed in real-time and with high-resolution. Hyperspectral snapshot mosaic sensors offer a promising approach due to their fast acquisition speed and compact size. However, a demosaicking algorithm is required to fully recover the spatial and spectral information of the snapshot images. Most state-of-the-art demosaicking algorithms require ground-truth training data with paired snapshot and high-resolution hyperspectral images, but such imagery pairs with the exact same scene are physically impossible to acquire in intraoperative settings. In this work, we present a fully unsupervised hyperspectral image demosaicking algorithm which only requires exemplar snapshot images for training purposes. We regard hyperspectral demosaicking as an ill-posed linear inverse problem which we solve using a deep neural network. We take advantage of the spectral correlation occurring in natural scenes to design a novel inter spectral band regularisation term based on spatial gradient consistency. By combining our proposed term with standard regularisation techniques and exploiting a standard data fidelity term, we obtain an unsupervised loss function for training deep neural networks, which allows us to achieve real-time hyperspectral image demosaicking. Quantitative results on hyperspetral image datasets show that our unsupervised demosaicking approach can achieve similar performance to its supervised counter-part, and significantly outperform linear demosaicking. A qualitative user study on real snapshot hyperspectral surgical images confirms the results from the quantitative analysis. Our results suggest that the proposed unsupervised algorithm can achieve promising hyperspectral demosaicking in real-time thus advancing the suitability of the modality for intraoperative use.

A Snapshot Near-Infrared Hyperspectral Demosaicing Method with Convolutional Neural Networks in Low Illumination Environment

A Snapshot Multi-Spectral Demosaicing Method for Multi-Spectral Filter Array Images Based on Channel Attention Network

Fast non-iterative blind restoration of hyperspectral images with PSFs

A Plug-and-play Demosaicing and Denoising Method on Mobile Cameras

Hyperspectral Imaging from a Raw Mosaic Image with End-to-end Learning

PPI Edge Infused Spatial-Spectral Adaptive Residual Network for Multispectral Filter Array Image Demosaicing

Efficient and Accurate Hyperspectral Image Demosaicing with Neural Network Architectures

A hyperspectral image denoising method based on land cover spectral autocorrelation

Unsupervised Spectral Demosaicing with Lightweight Spectral Attention Networks

DemosaicFormer: Coarse-to-Fine Demosaicing Network for HybridEVS Camera

TV-enhanced Deep Unfolding Network for Multispectral Image Demosaicing

Mosaic Convolution-Attention Network for Demosaicing Multispectral Filter Array Images

Learning Hyperspectral Images from RGB Images Via a Coarse-to-fine CNN.

ADMM-DSP: A Deep Spectral Image Prior for Snapshot Spectral Image Demosaicing

Hyperspectral Image Super-Resolution Via Convolutional Neural Network

NTIRE 2022 Spectral Demosaicing Challenge and Data Set

Channel-by-Channel Demosaicking Networks with Embedded Spectral Correlation

Deep Color Image Demosaicking with Feature Pyramid Channel Attention

Spatial gradient consistency for unsupervised learning of hyperspectral demosaicking: Application to surgical imaging

Deep learning approach for hyperspectral image demosaicking, spectral correction and high-resolution RGB reconstruction

Event Camera Demosaicing via Swin Transformer and Pixel-focus Loss