Abstract:Multispectral image (MS) and panchromatic image (PAN) fusion, which is also named as multispectral pansharpening, aims to obtain MS with high spatial resolution and high spectral resolution. However, due to the usual neglect of noise and blur generated in the imaging and transmission phases of data during training, many deep learning (DL) pansharpening methods fail to perform on the dataset containing noise and blur. To tackle this problem, a variational optimization-guided two-stage network (VOGTNet) for multispectral pansharpening is proposed in this work, and the performance of variational optimization (VO)-based pansharpening methods relies on prior information and estimates of spatial-spectral degradation from the target image to other two original images. Concretely, we propose a dual-branch fusion network (DBFN) based on supervised learning and train it by using the datasets containing noise and blur to generate the prior fusion result as the prior information that can remove noise and blur in the initial stage. Subsequently, we exploit the estimated spectral response function (SRF) and point spread function (PSF) to simulate the process of spatial-spectral degradation, respectively, thereby making the prior fusion result and the adaptive recovery model (ARM) jointly perform unsupervised learning on the original dataset to restore more image details and results in the generation of the high-resolution MSs in the second stage. Experimental results indicate that the proposed VOGTNet improves pansharpening performance and shows strong robustness against noise and blur. Furthermore, the proposed VOGTNet can be extended to be a general pansharpening framework, which can improve the ability to resist noise and blur of other supervised learning-based pansharpening methods. The source code is available at https://github.com/HZC-1998/VOGTNet.

A New Variational Approach Based on Proximal Deep Injection and Gradient Intensity Similarity for Spatio-Spectral Image Fusion

VO+Net: an Adaptive Approach Using Variational Optimization and Deep Learning for Panchromatic Sharpening

A Hybrid Pan-Sharpening Approach Using Nonnegative Matrix Factorization for WorldView Imageries.

A New Context-Aware Details Injection Fidelity With Adaptive Coefficients Estimation for Variational Pansharpening

Variational Zero-shot Multispectral Pansharpening

Deep Image Fusion Accounting for Inter-Image Variability.

An Iterative Approach for Image Fusion with Dynamic Gradient Sparsity and Anisotropic Spectral–spatial Total Variation

Variational Pansharpening Based on Coefficient Estimation With Nonlocal Regression

Pan-sharpening: a Fast Variational Fusion Approach

Spatial–Spectral Fusion by Combining Deep Learning and Variational Model

VOGTNet: Variational Optimization-Guided Two-Stage Network for Multispectral and Panchromatic Image Fusion

The Fusion of Panchromatic and Multispectral Remote Sensing Images Via Tensor-Based Sparse Modeling and Hyper-Laplacian Prior

Image Fusion Via Dynamic Gradient Sparsity and Anisotropic Spectral-Spatial Total Variation

Hyperspectral Image Fusion and Multitemporal Image Fusion by Joint Sparsity

Pan-sharpening of multi-spectral images using a new variational model

Joint Spatial-Spectral Optimization for the High-Magnification Fusion of Hyperspectral and Multispectral Images

Spatial-Spectral Fusion by Combining Deep Learning and Variation Model

MESSFN : a Multi-level and Enhanced Spectral-Spatial Fusion Network for Pan-sharpening

DMF-Net: A Dual Remote Sensing Image Fusion Network Based on Multi-Scale Convolutional Dense Connectivity with Performance Measure

Variational Model-Based Very High Spatial Resolution Remote Sensing Image Fusion

A Variational Pan-Sharpening with Local Gradient Constraints.