Abstract:Significance: Single-chip imaging devices featuring vertically stacked photodiodes and pixelated spectral filters are advancing multi-dye imaging methods for cancer surgeries, though this innovation comes with a compromise in spatial resolution. To mitigate this drawback, we developed a deep convolutional neural network (CNN) aimed at demosaicing the color and near-infrared (NIR) channels, with its performance validated on both pre-clinical and clinical datasets. Aim: We introduce an optimized deep CNN designed for demosaicing both color and NIR images obtained using a hexachromatic imaging sensor. Approach: A residual CNN was fine-tuned and trained on a dataset of color images and subsequently assessed on a series of dual-channel, color, and NIR images to demonstrate its enhanced performance compared with traditional bilinear interpolation. Results: Our optimized CNN for demosaicing color and NIR images achieves a reduction in the mean square error by 37% for color and 40% for NIR, respectively, and enhances the structural dissimilarity index by 37% across both imaging modalities in pre-clinical data. In clinical datasets, the network improves the mean square error by 35% in color images and 42% in NIR images while enhancing the structural dissimilarity index by 39% in both imaging modalities. Conclusions: We showcase enhancements in image resolution for both color and NIR modalities through the use of an optimized CNN tailored for a hexachromatic image sensor. With the ongoing advancements in graphics card computational power, our approach delivers significant improvements in resolution that are feasible for real-time execution in surgical environments.

Convolutional Sparse Coding for RGB+NIR Imaging.

Fusion of Low-Illuminance Visible and Near-Infrared Images Based on Convolutional Neural Networks

Convolutional Neural Network Based Computational Imaging Spectroscopy

Compressive Single-Pixel Hyperspectral Imaging Using RGB Sensors

Simultaneous Coded Aperture and Dictionary Optimization in Compressive Spectral Imaging Via Coherence Minimization.

Region-aware RGB and Near-Infrared Image Fusion

Color Router-based Long-wave Infrared Multispectral Imaging

Multispectral Fusion of RGB and NIR Images Using Weighted Least Squares and Convolution Neural Networks

Hybrid Fusion and Interpolation Algorithm with Near-Infrared Image

Hybrid Fusion And Demosaicing Algorithm With Near-Infrared Image

CCSR-Net: Unfolding Coupled Convolutional Sparse Representation for Multi-focus Image Fusion.

Single-image-based Nonuniformity Correction of Uncooled Long-Wave Infrared Detectors: a Deep-Learning Approach.

Fusion of Infrared and Visible Images Via Multi-Layer Convolutional Sparse Representation

Wavelength encoding spectral imaging based on the combination of deeply learned filters and an RGB camera

Color Night Vision for Navigation and Surveillance

An Integrated Enhancement Solution for 24-hour Colorful Imaging

Optimal LED Spectral Multiplexing for NIR2RGB Translation

Convolutional neural network advances in demosaicing for fluorescent cancer imaging with color-near-infrared sensors

Deep-learning Enables Single-Pixel Spectral Imaging

Multi-spectral compressive snapshot imaging using RGB image sensors

A robust infrared and visible image fusion framework via multi-receptive-field attention and color visual perception