Multi-scale hyperspectral recovery networks: RGB-hyperspectral imaging consistency empowered deep spectral super-resolution

Chen Li,Zhibo Xiao,Shouyu Wang

DOI: https://doi.org/10.1364/oe.525732

IF: 3.8

2024-06-12

Optics Express

Abstract:Chen Li, Zhibo Xiao, Shouyu Wang Hyperspectral imaging, which analyzes a spectrum of light instead of primary colors, can provide rich physical and chemical information not ... [Opt. Express 32, 23392-23403 (2024)]

optics

What problem does this paper attempt to address?

Compressive Single-Pixel Hyperspectral Imaging Using RGB Sensors

Chenning Tao,Huanzheng Zhu,Xucheng Wang,Shuhang Zheng,Qin Xie,Chang Wang,Rengmao Wu,Zhenrong Zheng

DOI: https://doi.org/10.1364/oe.416388

IF: 3.8

2021-01-01

Optics Express

Abstract:Hyperspectral imaging that obtains the spatial-spectral information of a scene has been extensively applied in various fields but usually requires a complex and costly system. A single-pixel detector based hyperspectral system mitigates the complexity problem but simultaneously brings new difficulties on the spectral dispersion device. In this work, we propose a low-cost compressive single-pixel hyperspectral imaging system with RGB sensors. Based on the structured illumination single-pixel imaging configuration, the lens-free system directly captures data by the RGB sensors without dispersion in the spectral dimension. The reconstruction is performed with a pre-trained spatial-spectral dictionary, and the hyperspectral images are obtained through compressive sensing. In addition, the spatial patterns for the structured illumination and the dictionary for the sparse representation are optimized by coherence minimization, which further improve the reconstruction quality. In both spatial and spectral dimensions, the intrinsic sparse properties of the hyperspectral images are made full use of for high sampling efficiency and low reconstruction cost. This work may introduce opportunities for optimization of computational imaging systems and reconstruction algorithms towards high speed, high resolution, and low cost future.
Multispectral Image Super-Resolution Via RGB Image Fusion and Radiometric Calibration

Zhi-Wei Pan,Hui-Liang Shen

DOI: https://doi.org/10.1109/tip.2018.2881911

IF: 10.6

2019-01-01

IEEE Transactions on Image Processing

Abstract:Multispectral imaging is of wide application for its capability in acquiring the spectral information of scenes. Due to hardware limitation, multispectral imaging device usually cannot achieve high-spatial resolution. To address the issue, this paper proposes a multispectral image super-resolution algorithm, referred as SRIF, by fusing the low-resolution multispectral image and the high-resolution (HR) RGB image. It deals with the general circumstance that image intensity is linear to scene radiance for multispectral imaging devices while is nonlinear and unknown for most RGB cameras. The SRIF algorithm first solves the inverse camera response function and a spectral sensitivity function of RGB camera, and establishes the linear relationship between multispectral and RGB images. Then the unknown HR multispectral image is efficiently reconstructed according to the linear image degradation models. Meanwhile, the edge structure of the reconstructed HR multispectral image is kept in accordance with that of the RGB image using a weighted total variation regularizer. The effectiveness of the SRIF algorithm is evaluated on both public datasets and our image set. Experimental results validate that the SRIF algorithm outperforms the state-of-the-arts in terms of both reconstruction accuracy and computational efficiency.
Superpixel-Based Linear Reconstruction Method for Dual-Camera Compressed Hyperspectral Imaging System

Bingliang Chen,Yang Zhang,Zeqing Yu,Xinyu Liu,Qiangbo Zhang,Chang Wang,Zhenrong Zheng

DOI: https://doi.org/10.1016/j.optcom.2023.129618

IF: 2.4

2023-01-01

Optics Communications

Abstract:Compressive spectral imaging systems have received significant research attention in recent years. Among these systems, the dual-camera compressive hyperspectral imaging (DCCHI) design, has the potential to improve imaging resolution while maintaining the snapshot advantage. In this paper, we propose superpixel-based linear reconstruction methods for DCCHI system. Specifically, we assume that the spectral intensity of the superpixels generated from the RGB image varies linearly in the spectral dimension. Instead of addressing an ill-posed problem, we reconstruct hyperspectral images via solving overdetermined equations using the least squares method, resulting in a significant reduction in both computation time and memory usage. To further enhance the reconstruction accuracy, we extend the spectra of superpixels from a 1D linear space into a 3D space with an RGB spectral response matrix. Despite the simplicity of our method, it offers improved reconstruction accuracy of hyperspectral images compared to other state-of-the-art methods. Furthermore, it is more efficient in terms of memory usage and computational time.
Hyperspectral image recovery based on fusion of coded aperture snapshot spectral imaging and RGB images by guided filtering

Chenning Tao,Huanzheng Zhu,Peng Sun,Rengmao Wu,Zhenrong Zheng

DOI: https://doi.org/10.1016/j.optcom.2019.124804

IF: 2.4

2020-01-01

Optics Communications

Abstract:In compressive hyperspectral imaging systems, the key issue is to accurately and efficiently recover 3D spectral data cubes from spectrally undersampled 2D projections. In this paper, a guided-filtering-based reconstruction algorithm is proposed to recover hyperspectral data cube with a joint coded aperture snapshot spectral imaging (CASSI) and RGB imaging system. In order to preserve the spectral and spatial details in the CASSI and the RGB branch to the greatest extent, 2D projections from the two branches are reconstructed into hyperspectral images individually with a pre-trained dictionary. The appropriate combination of the information from the two branches is realized through guided image filtering, which is applied on the reconstruction result of the CASSI branch with the result from the RGB branch as the guidance image. In virtue of combination between the well-performed spectral fidelity of the CASSI branch and the high spatial accuracy of the RGB branch, the proposed guided-filtering-based compressive hyperspectral imaging (GFCHI) not only overcomes the block effect and blurring common in the state-of-the-art methods, but also improves spectral fidelity related to key applications of hyperspectral imaging. The quantitative metrics associated with spatial, spectral and overall performance demonstrate the state-of-the-art reconstruction quality of GFCHI.
Accurate Spectral Super-resolution from Single RGB Image Using Multi-scale CNN

Yiqi Yan,Lei Zhang,Jun Li,Wei Wei,Yanning Zhang

DOI: https://doi.org/10.48550/arXiv.1806.03575

2018-06-10

Computer Vision and Pattern Recognition

Abstract:Different from traditional hyperspectral super-resolution approaches that focus on improving the spatial resolution, spectral super-resolution aims at producing a high-resolution hyperspectral image from the RGB observation with super-resolution in spectral domain. However, it is challenging to accurately reconstruct a high-dimensional continuous spectrum from three discrete intensity values at each pixel, since too much information is lost during the procedure where the latent hyperspectral image is downsampled (e.g., with x10 scaling factor) in spectral domain to produce an RGB observation. To address this problem, we present a multi-scale deep convolutional neural network (CNN) to explicitly map the input RGB image into a hyperspectral image. Through symmetrically downsampling and upsampling the intermediate feature maps in a cascading paradigm, the local and non-local image information can be jointly encoded for spectral representation, ultimately improving the spectral reconstruction accuracy. Extensive experiments on a large hyperspectral dataset demonstrate the effectiveness of the proposed method.
HSCNN+: Advanced CNN-Based Hyperspectral Recovery from RGB Images

Zhan Shi,Chang Chen,Zhiwei Xiong,Dong Liu,Feng Wu

DOI: https://doi.org/10.1109/CVPRW.2018.00139

2018-01-01

Abstract:Hyperspectral recovery from a single RGB image has seen a great improvement with the development of deep convolutional neural networks (CNNs). In this paper, we propose two advanced CNNs for the hyperspectral reconstruction task, collectively called HSCNN+. We first develop a deep residual network named HSCNN-R, which comprises a number of residual blocks. The superior performance of this model comes from the modern architecture and optimization by removing the hand-crafted upsampling in HSCNN. Based on the promising results of HSCNN-R, we propose another distinct architecture that replaces the residual block by the dense block with a novel fusion scheme, leading to a new network named HSCNN-D. This model substantially deepens the network structure for a more accurate solution. Experimental results demonstrate that our proposed models significantly advance the state-of-the-art. In the NTIRE 2018 Spectral Reconstruction Challenge, our entries rank the 1st (HSCNN-D) and 2nd (HSCNN-R) places on both the "Clean" and "Real World" tracks. (Codes are available at [clean-r], [realworld-r], [clean-d], and [realworld-d]).
Spectral Representation Via Data-Guided Sparsity for Hyperspectral Image Super-Resolution

Xian-Hua Han,Yongqing Sun,Jian Wang,Boxin Shi,Yinqiang Zheng,Yen-Wei Chen

DOI: https://doi.org/10.3390/s19245401

IF: 3.9

2019-01-01

Sensors

Abstract:Hyperspectral imaging is capable of acquiring the rich spectral information of scenes and has great potential for understanding the characteristics of different materials in many applications ranging from remote sensing to medical imaging. However, due to hardware limitations, the existed hyper-/multi-spectral imaging devices usually cannot obtain high spatial resolution. This study aims to generate a high resolution hyperspectral image according to the available low resolution hyperspectral and high resolution RGB images. We propose a novel hyperspectral image superresolution method via non-negative sparse representation of reflectance spectra with a data guided sparsity constraint. The proposed method firstly learns the hyperspectral dictionary from the low resolution hyperspectral image and then transforms it into the RGB one with the camera response function, which is decided by the physical property of the RGB imaging camera. Given the RGB vector and the RGB dictionary, the sparse representation of each pixel in the high resolution image is calculated with the guidance of a sparsity map, which measures pixel material purity. The sparsity map is generated by analyzing the local content similarity of a focused pixel in the available high resolution RGB image and quantifying the spectral mixing degree motivated by the fact that the pixel spectrum of a pure material should have sparse representation of the spectral dictionary. Since the proposed method adaptively adjusts the sparsity in the spectral representation based on the local content of the available high resolution RGB image, it can produce more robust spectral representation for recovering the target high resolution hyperspectral image. Comprehensive experiments on two public hyperspectral datasets and three real remote sensing images validate that the proposed method achieves promising performances compared to the existing state-of-the-art methods.
Spectral Reflectance Recovery From a Single RGB Image

Ying Fu,Yongrong Zheng,Lin Zhang,Hua Huang

DOI: https://doi.org/10.1109/tci.2018.2855445

IF: 5.4

2018-09-01

IEEE Transactions on Computational Imaging

Abstract:The spectral reflectance is an intrinsic and discriminative characteristic of object materials and can be obtained by hyperspectral imaging. However, existing hyperspectral cameras are limited in low-spatial/temporal resolution while yet being complicated and expensive. In this paper, we present a nonnegative sparse representation based method to recover high-quality spectral reflectance from a single RGB image. Unlike previous methods, our approach learns multiple nonnegative sparse coding dictionaries from the training spectral dataset in terms of clustering results. Then, the spectral reflectance of the input RGB image is recovered based on nonnegative sparse representation, which also considers the spatial structured similarity and high correlation across spectra under the learned dictionaries. Furthermore, the illumination spectrum can be estimated with the recovered spectral reflectance under the known RGB camera spectral sensitivity. Experimental results show that the proposed method outperforms state-of-the-art spectral reflectance recovery methods in terms of both objective metrics and subjective visual quality. Besides, we show an application of our method to accurately relight scenes under the novel illumination.

engineering, electrical & electronic,imaging science & photographic technology
Hyperspectral Image Super-Resolution Using Sparse Spectral Unmixing and Low-Rank Constraints

Zeyu Li,Chao Li,Cheng Deng,Jie Li

DOI: https://doi.org/10.1109/igarss.2016.7730884

2016-01-01

Abstract:Hyperspectral images play an important role in real-world applications, such as recognition and remote sensing, etc. How to enhance the spatial resolution of hyperspectral image is still a challenging problem in this field. In this paper, we propose a novel hyperspectral image super-resolution approach by jointly incorporating the sparse, low-rank constraints and spectral mixture priori into a linear unmixing framework, which will make the unmixing framework more consistent with the real-world scenarios of the spectral mixture. Experiments on two public databases show that our proposed approach achieves much lower average reconstruction errors than other state-of-the-art methods.
Remote Sensing Hyperspectral Image Super-Resolution via Multidomain Spatial Information and Multiscale Spectral Information Fusion

Chi Chen,Yongcheng Wang,Yuxi Zhang,Zhikang Zhao,Hao Feng

DOI: https://doi.org/10.1109/tgrs.2024.3388531

IF: 8.2

2024-04-27

IEEE Transactions on Geoscience and Remote Sensing

Abstract:Hyperspectral image (HSI) super-resolution (SR) technology has made remarkable progress due to the development of deep learning (DL). However, the technique still faces two challenges, i.e., the imbalance between spectral and spatial information extraction, and the parameter deviation and high computational effort associated with 3-D convolution. In this article, we propose an SR method for remote sensing HSIs based on multidomain spatial information and multiscale spectral information fusion (MSSR). Specifically, inspired by the high degree of self-similarity of remote sensing HSIs, a spatial–spectral attention module based on dilated convolution (DSSA) for capturing global spatial information is proposed. The extraction of local spatial information is then accomplished by residual blocks using small-size convolution kernels. Meanwhile, we propose the 3-D inception module to efficiently mine multiscale spectral information. The module only retains the scale of the 3-D convolution kernel in spectral dimension, which greatly reduces the high computational cost caused by 3-D convolution. Comparative experimental results on four benchmark datasets demonstrate that compared with the current cutting-edge models, our method achieves state-of-the-art (SOTA) results and the model computation is greatly reduced.

imaging science & photographic technology,remote sensing,engineering, electrical & electronic,geochemistry & geophysics
HSCNN: CNN-Based Hyperspectral Image Recovery from Spectrally Undersampled Projections

Zhiwei Xiong,Zhan Shi,Huiqun Li,Lizhi Wang,Dong Liu,Feng Wu

DOI: https://doi.org/10.1109/iccvw.2017.68

2017-01-01

Abstract:This paper presents a unified deep learning framework to recover hyperspectral images from spectrally undersampled projections. Specifically, we investigate two kinds of representative projections, RGB and compressive sensing (CS) measurements. These measurements are first upsampled in the spectral dimension through simple interpolation or CS reconstruction, and the proposed method learns an end-to-end mapping from a large number of upsampled/groundtruth hyperspectral image pairs. The mapping is represented as a deep convolutional neural network (CNN) that takes the spectrally upsampled image as input and outputs the enhanced hyperspetral one. We explore different network configurations to achieve high reconstruction fidelity. Experimental results on a variety of test images demonstrate significantly improved performance of the proposed method over the state-of-the-arts.
Difference Curvature Multidimensional Network for Hyperspectral Image Super-Resolution

Chi Zhang,Mingjin Zhang,Yunsong Li,Xinbo Gao,Shi Qiu

DOI: https://doi.org/10.3390/rs13173455

IF: 5

2021-08-31

Remote Sensing

Abstract:In recent years, convolutional-neural-network-based methods have been introduced to the field of hyperspectral image super-resolution following their great success in the field of RGB image super-resolution. However, hyperspectral images appear different from RGB images in that they have high dimensionality, implying a redundancy in the high-dimensional space. Existing approaches struggle in learning the spectral correlation and spatial priors, leading to inferior performance. In this paper, we present a difference curvature multidimensional network for hyperspectral image super-resolution that exploits the spectral correlation to help improve the spatial resolution. Specifically, we introduce a multidimensional enhanced convolution (MEC) unit into the network to learn the spectral correlation through a self-attention mechanism. Meanwhile, it reduces the redundancy in the spectral dimension via a bottleneck projection to condense useful spectral features and reduce computations. To remove the unrelated information in high-dimensional space and extract the delicate texture features of a hyperspectral image, we design an additional difference curvature branch (DCB), which works as an edge indicator to fully preserve the texture information and eliminate the unwanted noise. Experiments on three publicly available datasets demonstrate that the proposed method can recover sharper images with minimal spectral distortion compared to state-of-the-art methods. PSNR/SAM is 0.3–0.5 dB/0.2–0.4 better than the second best methods.

environmental sciences,imaging science & photographic technology,remote sensing,geosciences, multidisciplinary
Hyper-Spectral Image Super-Resolution Using Non-Negative Spectral Representation with Data-Guided Sparsity

Xian-Hua Han,Jian Wang,Boxin Shi,YinQiang Zheng,Yen-Wei Chen

DOI: https://doi.org/10.1109/ism.2017.99

2017-01-01

Abstract:Hyperspectral imaging has great potential for understanding the characteristics of different materials in many applications ranging from remote sensing to medical imaging. However, due to various hardware limitations, only low-resolution hyperspectral and high-resolution multi-spectral images can be available using existing imaging techniques. This study aims to generate a high-resolution hyperspectral image via fusion of the available LR-HS and HR-MS images. We propose a novel hyperspectral image superresolution method via non-negative sparse representation of reflectance spectral with adaptive sparsity constraint. By analyzing local content similarity of a focused pixel in the available high-resolution multi-spectral image, which can measure pixel material purity according to surrounding pixels, we generate a sparsity map for guiding non-negative sparse coding optimization procedure of the spectral representation called non-negative spectral representation with data guided sparsity. Since the proposed method adaptively adjust the sparsity in the spectral representation based on the local content of the available high-resolution multi-spectral image, it can produce more robust spectral representation for recovering the target high-resolution hyper-spectral image. Comprehensive experiments on two public hyperspectral datasets validate that the proposed method achieves promising performances compared with the existing state of the art methods.
Hyperspectral Image Super-Resolution Based on Feature Diversity Extraction

Jing Zhang,Renjie Zheng,Zekang Wan,Ruijing Geng,Yi Wang,Yu Yang,Xuepeng Zhang,Yunsong Li

DOI: https://doi.org/10.3390/rs16030436

IF: 5

2024-01-23

Remote Sensing

Abstract:Deep learning is an important research topic in the field of image super-resolution. Problematically, the performance of existing hyperspectral image super-resolution networks is limited by feature learning for hyperspectral images. Nevertheless, the current algorithms exhibit some limitations in extracting diverse features. In this paper, we address limitations to existing hyperspectral image super-resolution networks, focusing on feature learning challenges. We introduce the Channel-Attention-Based Spatial–Spectral Feature Extraction network (CSSFENet) to enhance hyperspectral image feature diversity and optimize network loss functions. Our contributions include: (a) a convolutional neural network super-resolution algorithm incorporating diverse feature extraction to enhance the network's diversity feature learning by elevating the matrix rank, (b) a three-dimensional (3D) feature extraction convolution module, the Channel-Attention-Based Spatial–Spectral Feature Extraction Module (CSSFEM), to boost the network's performance in both the spatial and spectral domains, (c) a feature diversity loss function designed based on the image matrix's singular value to maximize element independence, and (d) a spatial–spectral gradient loss function introduced based on space and spectrum gradient values to enhance the reconstructed image's spatial–spectral smoothness. In contrast to existing hyperspectral super-resolution algorithms, we used four evaluation indexes, PSNR, mPSNR, SSIM, and SAM, and our method showed superiority during testing with three common hyperspectral datasets.

environmental sciences,imaging science & photographic technology,remote sensing,geosciences, multidisciplinary
Unsupervised Across Domain Consistency- Difference Network for Hyperspectral Image Super-Resolution

Zhiling Guo,Jingwei Xin,Nannan Wang,Jie Li,Xiaoyu Wang,Xinbo Gao

DOI: https://doi.org/10.1109/tgrs.2023.3275135

IF: 8.2

2023-05-26

IEEE Transactions on Geoscience and Remote Sensing

Abstract:Without reducing the spectral resolution, hyperspectral image (HSI) super-resolution has achieved remarkable progress thanks to the success of deep neural networks. However, existing methods cannot fully excavate the latent high-frequency details only in the single spatial domain. Different from existing methods that only achieves the super-resolution task in spatial domain, we optimize the amplitude spectrum and phase spectrum in frequency domain to obtain high-resolution HSI (HR-HSI). We propose a new unsupervised framework to reconstruct HR-HSI using only the observed low resolution (LR) HSI and HR multispectral image (MSI). Based on triple-level modeling, the encoder–decoder learns abundant features including contextual information from multiple scales. In addition, we propose iterative across domain consistency-difference (ADCD) module, which is embedded between encoder and decoder. In ADCD module, three parallel convolution streams (amplitude spectrum adjustment branch, phase spectrum adjustment branch and spatial domain branch) are used to explore the consistency-difference between each other, which is preserved by memory units within the module. Particularly, we embed the dilated causal convolution in the frequency domain processing branch, which is convenient to flexibly adjust the receptive field and adapt to different domains. Extensive experiments are conducted on widely used datasets in comparison with state-of-the-art models, demonstrating the advantage of the proposed method.

imaging science & photographic technology,remote sensing,engineering, electrical & electronic,geochemistry & geophysics
A Recurrent Feedback Hyperspectral Image Super-Resolution Reconstruction Method by Using Self-Attention-Based Pixel Awareness

Ruyi Feng,Zhongyu Guo,Xiaofeng Wang

DOI: https://doi.org/10.1109/jstars.2024.3471899

IF: 4.715

2024-10-25

IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing

Abstract:Hyperspectral images (HSIs) contain abundant spectral information, while the spatial resolution is usually limited. To obtain high-spatial-resolution HSIs, various HSI super-resolution (SR) methods are proposed. Currently, deep-learning-based SR reconstruction methods are studied in depth, which take different measures to make full use of the spatial and spectral information of HSIs, and optimize the network with lots of trainings. Although they have achieved satisfying spatial resolution, the spectral consistency before and after reconstruction is difficult to guarantee. In this article, we proposed a self-attention-based recurrent feedback network for hyperspectral SR reconstruction, utilizing pixel-aware weights and pseudo three-dimensional convolution to enhance the spatial and spectral consistency during the reconstruction process. In addition, group reconstruction is used to reduce the redundancy of information. Spectral consistency regularization is proposed to ensure the spectral consistency before and after reconstruction. The effectiveness of the proposed method is tested on one set of natural images and three hyperspectral remote sensing image datasets.

imaging science & photographic technology,remote sensing,engineering, electrical & electronic,geography, physical
Multi-scale spatial-spectral Transformer for spectral reconstruction from RGB images

HongKang LuanYuBao SunKaiXuan HuangQuan GuDigital Forensics Engineering Research Center of the Ministry of Education,Jiangsu Collaborative Innovation Center of Atmospheric Environment and Equipment Technology,Nanjing University of Information Science and Technology,Nanjing,China

DOI: https://doi.org/10.1080/01431161.2023.2295831

IF: 3.531

2024-01-08

International Journal of Remote Sensing

Abstract:Spectrometers can obtain fine spectral reflectance of substances through high-resolution spectral sampling, which has important advantages in identifying the categories of ground substances. As a complement to high-cost spectrometers, reconstruction from RGB images provides an economical and convenient way to acquire hyperspectral images. Although the current mainstream reconstruction methods based on deep convolution network can directly learn the inverse reconstruction mapping in data-driven way, they still have shortcomings in the representation of multi-scale and long-range spatial-spectral correlations within hyperspectral images. To conquer these issues, we propose a novel multi-scale spatial-spectral Transformer network for spectral reconstruction. The proposed network consists of a cascade of multiple-scale spatial contextual module and spatial-spectral fusion Transformer module. Specifically, the multiple-scale spatial contextual module performs three-level feature extraction to learn the spatial contextual structures of each spectral band from the RGB image. The spatial-spectral fusion Transformer module employs three parallel spatial-spectral united Transformers and one fusion Transformer to enhance the spatial and spectral consistency of the reconstructed spectral image. Parallel deployment of spatial-spectral united Transformers is beneficial to reduce the over-smooth effect caused by stacking too many Transformers. Therefore, the proposed network can better recover hyperspectral images with both fine spatial structures and accurate spectral signatures. Comprehensive experiments show that our method can achieve better reconstruction performance than the state-of-the-art methods.

imaging science & photographic technology,remote sensing
Hyperspectral Image Super-resolution via Deep Spatio-spectral Convolutional Neural Networks

Jin-Fan Hu,Ting-Zhu Huang,Liang-Jian Deng,Tai-Xiang Jiang,Gemine Vivone,Jocelyn Chanussot

DOI: https://doi.org/10.48550/arXiv.2005.14400

2020-05-29

Abstract:Hyperspectral images are of crucial importance in order to better understand features of different materials. To reach this goal, they leverage on a high number of spectral bands. However, this interesting characteristic is often paid by a reduced spatial resolution compared with traditional multispectral image systems. In order to alleviate this issue, in this work, we propose a simple and efficient architecture for deep convolutional neural networks to fuse a low-resolution hyperspectral image (LR-HSI) and a high-resolution multispectral image (HR-MSI), yielding a high-resolution hyperspectral image (HR-HSI). The network is designed to preserve both spatial and spectral information thanks to an architecture from two folds: one is to utilize the HR-HSI at a different scale to get an output with a satisfied spectral preservation; another one is to apply concepts of multi-resolution analysis to extract high-frequency information, aiming to output high quality spatial details. Finally, a plain mean squared error loss function is used to measure the performance during the training. Extensive experiments demonstrate that the proposed network architecture achieves best performance (both qualitatively and quantitatively) compared with recent state-of-the-art hyperspectral image super-resolution approaches. Moreover, other significant advantages can be pointed out by the use of the proposed approach, such as, a better network generalization ability, a limited computational burden, and a robustness with respect to the number of training samples.

Image and Video Processing,Computer Vision and Pattern Recognition
Hybrid 2-D–3-D Deep Residual Attentional Network With Structure Tensor Constraints for Spectral Super-Resolution of RGB Images

Jiaojiao Li,Chaoxiong Wu,Rui Song,Weiying Xie,Chiru Ge,Bo Li,Yunsong Li

DOI: https://doi.org/10.1109/tgrs.2020.3004934

IF: 8.2

2021-03-01

IEEE Transactions on Geoscience and Remote Sensing

Abstract:RGB image spectral super-resolution (SSR) is a challenging task due to its serious ill-posedness, which aims at recovering a hyperspectral image (HSI) from a corresponding RGB image. In this article, we propose a novel hybrid 2-D–3-D deep residual attentional network (HDRAN) with structure tensor constraints, which can take fully advantage of the spatial–spectral context information in the reconstruction progress. Previous works improve the SSR performance only through stacking more layers to catch local spatial correlation neglecting the differences and interdependences among features, especially band features; different from them, our novel method focuses on the context information utilization. First, the proposed HDRAN consists of a 2D-RAN following by a 3D-RAN, where the 2D-RAN mainly focuses on extracting abundant spatial features, whereas the 3D-RAN mainly simulates the interband correlations. Then, we introduce 2-D channel attention and 3-D band attention mechanisms into the 2D-RAN and 3D-RAN, respectively, to adaptively recalibrate channelwise and bandwise feature responses for enhancing context features. Besides, since structure tensor represents structure and spatial information, we apply structure tensor constraint to further reconstruct more accurate high-frequency details during the training process. Experimental results demonstrate that our proposed method achieves the state-of-the-art performance in terms of mean relative absolute error (MRAE) and root mean square error (RMSE) on both the "clean" and "real world" tracks in the NTIRE 2018 Spectral Reconstruction Challenge. As for competitive ranking metric MRAE, our method separately achieves a 16.06% and 2.90% relative reduction on two tracks over the first place. Furthermore, we investigate HDRAN on the other two HSI benchmarks noted as t-e CAVE and Harvard data sets, also demonstrating better results than state-of-the-art methods.

imaging science & photographic technology,remote sensing,engineering, electrical & electronic,geochemistry & geophysics
A General Deep Learning Point–Surface Fusion Framework for RGB Image Super-Resolution

Yan Zhang,Lifu Zhang,Ruoxi Song,Qingxi Tong

DOI: https://doi.org/10.3390/rs16010139

IF: 5

2024-01-01

Remote Sensing

Abstract:Hyperspectral images are usually acquired in a scanning-based way, which can cause inconvenience in some situations. In these cases, RGB image spectral super-resolution technology emerges as an alternative. However, current mainstream spectral super-resolution methods aim to generate continuous spectral information at a very narrow range, limited to the visible light range. Some researchers introduce hyperspectral images as auxiliary data. But it is usually required that the auxiliary hyperspectral images have the same spatial range as RGB images. To address this issue, a general point–surface data fusion method is designed to achieve the RGB image spectral super-resolution goal in this paper, named GRSS-Net. The proposed method utilizes hyperspectral point data as auxiliary data to provide spectral reference information. Thus, the spectral super-resolution can extend the spectral reconstruction range according to spectral data. The proposed method utilizes compressed sensing theory as a fundamental physical mechanism and then unfolds the traditional hyperspectral image reconstruction optimization problem into a deep network. Finally, a high-spatial-resolution hyperspectral image can be obtained. Thus, the proposed method combines the non-linear feature extraction ability of deep learning and the interpretability of traditional physical models simultaneously. A series of experiments demonstrates that the proposed method can effectively reconstruct spectral information in RGB images. Meanwhile, the proposed method provides a framework of spectral super-resolution for different applications.

Multi-scale hyperspectral recovery networks: RGB-hyperspectral imaging consistency empowered deep spectral super-resolution

Compressive Single-Pixel Hyperspectral Imaging Using RGB Sensors

Multispectral Image Super-Resolution Via RGB Image Fusion and Radiometric Calibration

Superpixel-Based Linear Reconstruction Method for Dual-Camera Compressed Hyperspectral Imaging System

Hyperspectral image recovery based on fusion of coded aperture snapshot spectral imaging and RGB images by guided filtering

Accurate Spectral Super-resolution from Single RGB Image Using Multi-scale CNN

HSCNN+: Advanced CNN-Based Hyperspectral Recovery from RGB Images

Spectral Representation Via Data-Guided Sparsity for Hyperspectral Image Super-Resolution

Spectral Reflectance Recovery From a Single RGB Image

Hyperspectral Image Super-Resolution Using Sparse Spectral Unmixing and Low-Rank Constraints

Remote Sensing Hyperspectral Image Super-Resolution via Multidomain Spatial Information and Multiscale Spectral Information Fusion

HSCNN: CNN-Based Hyperspectral Image Recovery from Spectrally Undersampled Projections

Difference Curvature Multidimensional Network for Hyperspectral Image Super-Resolution

Hyper-Spectral Image Super-Resolution Using Non-Negative Spectral Representation with Data-Guided Sparsity

Hyperspectral Image Super-Resolution Based on Feature Diversity Extraction

Unsupervised Across Domain Consistency- Difference Network for Hyperspectral Image Super-Resolution

A Recurrent Feedback Hyperspectral Image Super-Resolution Reconstruction Method by Using Self-Attention-Based Pixel Awareness

Multi-scale spatial-spectral Transformer for spectral reconstruction from RGB images

Hyperspectral Image Super-resolution via Deep Spatio-spectral Convolutional Neural Networks

Hybrid 2-D–3-D Deep Residual Attentional Network With Structure Tensor Constraints for Spectral Super-Resolution of RGB Images

A General Deep Learning Point–Surface Fusion Framework for RGB Image Super-Resolution