Abstract:Due to the physical boundaries, fusing low spatial resolution hyperspectral (LrHSI) with high spatial resolution multispectral (HrMSI) images is a hot and promising area for obtaining hyperspectral that have high spatial-spectral resolution images (HrHSI). Effectively formulating the fundamental features of hyperspectral images (HSI), such as global spectral correlation, nonlocal spatial correlation, as well spatial-spectral correlation, is complex in HSI-MSI fusion. Moreover, the fusion process is highly affected by the degradation systems, where these systems are not known in real scenarios. To this end, in this article, we proposed a model-guided deep unfolded fusion network with non-local spatial-spectral priors (MGDuNLSS-net) that can maintain the essential features of the HSIs and implicitly estimates the degradation process in an adequate running time. Specifically, the proposed method is designed based on subspace representation in an iterative manner and unrolling its steps toward a deep network as an end-to-end framework. This approach contains two submodules, fusion [nonlocal spatial-spectral block (NLSSB)] and imaging system submodules. The former submodule is proposed to exploit the images' intrinsic characteristics to improve the preservation of spectral and spatial details. NLSSB contains two nonlocal self-similarity (NLSS) layers embedded between two bidirectional simple recurrent unit (BSRU) layers. The recurrent calculation, as well as refined components to maintain the global spectral correlation, are the light recurrence operation and highway network, while 3-D convolutions in the BSRU can retain the spatial-spectral correlation. The NLSS layer can efficiently and effectively model long-range spatial contexts, which is designed based on criss-cross attention. The later submodule is used to refine the prediction of the degradation process at any iteration via backprojecting the estimated fused image to the observed pair, which can ensure the good performance of fusion. Compared with state-of-the-art fusion approaches, three remote sensing datasets are used to validate the proposed approach's performance.

Spatial–Spectral Fusion by Combining Deep Learning and Variational Model

Spatial-Spectral Fusion by Combining Deep Learning and Variation Model

Hyperspectral and Multispectral Image Fusion Based on Deep Attention Network.

Panchromatic and multi-spectral image fusion for new satellites based on multi-channel deep model

FusionNet: An Unsupervised Convolutional Variational Network for Hyperspectral and Multispectral Image Fusion

A Spatial–Spectral Dual-Optimization Model-Driven Deep Network for Hyperspectral and Multispectral Image Fusion

A Remote Sensing Spatiotemporal Fusion Model of Landsat and Modis Data Via Deep Learning

Explicit and Stepwise Models for Spatiotemporal Fusion of Remote Sensing Images with Deep Neural Networks

Learning Spectral Cues for Multispectral and Panchromatic Image Fusion

Spatiotemporal Fusion of Satellite Images Via Very Deep Convolutional Networks

A model-guided deep convolutional sparse coding network for hyperspectral and multispectral image fusion

Design of Intelligent Spectral Selection and Multispectral Imaging Fusion Algorithm Based on Deep Neural Network Model

Boosting the Accuracy of Multispectral Image Pansharpening by Learning a Deep Residual Network

FusionCNN: a Remote Sensing Image Fusion Algorithm Based on Deep Convolutional Neural Networks

A Novel Deep Learning-Based Spatiotemporal Fusion Method for Combining Satellite Images with Different Resolutions Using a Two-Stream Convolutional Neural Network

Hyperspectral and Multispectral Image Fusion by Deep Neural Network in a Self-Supervised Manner

Model-Guided Deep Unfolded Fusion Network with Nonlocal Spatial-Spectral Priors for Hyperspectral Image Super-Resolution.

A New Variational Approach Based on Proximal Deep Injection and Gradient Intensity Similarity for Spatio-Spectral Image Fusion

Spatio–Temporal–Spectral Collaborative Learning for Spatio–Temporal Fusion with Land Cover Changes

Deep-Learning-Based Spatio-Temporal-Spectral Integrated Fusion of Heterogeneous Remote Sensing Images

Review of Pixel-Level Remote Sensing Image Fusion Based on Deep Learning