Abstract:Recently, convolutional neural networks (CNN) have obtained promising results in single-image SR for hyperspectral pansharpening. However, enhancing CNNs' representation ability with fewer parameters and a shorter prediction time is a challenging and critical task. In this paper, we propose a novel multi-spectral image fusion method using a combination of the previously proposed 3D CNN model VolumeNet and 2.5D texture transfer method using other modality high resolution (HR) images. Since a multi-spectral (MS) image consists of several bands and each band is a 2D image slice, MS images can be seen as 3D data. Thus, we use the previously proposed VolumeNet to fuse HR panchromatic (PAN) images and bicubic interpolated MS images. Because the proposed 3D VolumeNet can effectively improve the accuracy by expanding the receptive field of the model, and due to its lightweight structure, we can achieve better performance against the existing method without purchasing a large number of remote sensing images for training. In addition, VolumeNet can restore the high-frequency information lost in the HR MR image as much as possible, reducing the difficulty of feature extraction in the following step: 2.5D texture transfer. As one of the latest technologies, deep learning-based texture transfer has been demonstrated to effectively and efficiently improve the visual performance and quality evaluation indicators of image reconstruction. Different from the texture transfer processing of RGB image, we use HR PAN images as the reference images and perform texture transfer for each frequency band of MS images, which is named 2.5D texture transfer. The experimental results show that the proposed method outperforms the existing methods in terms of objective accuracy assessment, method efficiency, and visual subjective evaluation.

MFT-GAN: A Multiscale Feature-Guided Transformer Network for Unsupervised Hyperspectral Pansharpening

SSETPAN: Spatial-Spectral Enhanced Transformer Based Network for Pansharpening

Multiscale 3-D-2-D Mixed CNN and Lightweight Attention-Free Transformer for Hyperspectral and LiDAR Classification

Mun-GAN: A Multiscale Unsupervised Network for Remote Sensing Image Pansharpening

Efficient and Accurate Hyperspectral Pansharpening Using 3D VolumeNet and 2.5D Texture Transfer

MFITN: A Multilevel Feature Interaction Transformer Network for Pansharpening

Pan-GAN: An unsupervised pan-sharpening method for remote sensing image fusion

An Unsupervised Generative Adversarial Network Based on U-Shaped Structure for Pansharpening

UPanGAN: Unsupervised pansharpening based on the spectral and spatial loss constrained Generative Adversarial Network

Transformer-Based Dual-Branch Multiscale Fusion Network for Pan-Sharpening Remote Sensing Images

Local-Global Based High-Resolution Spatial-Spectral Representation Network for Pansharpening

PanFormer: a Transformer Based Model for Pan-sharpening

Pan-sharpening Method Based on Unsupervised and Divide-and-conquer Strategies

Convolutional LSTM-Based Hierarchical Feature Fusion for Multispectral Pan-Sharpening

Self-Supervised Pansharpening Based on a Cycle-Consistent Generative Adversarial Network

PC-GANs: Progressive Compensation Generative Adversarial Networks for Pan-sharpening

Pan-Sharpening with Customized Transformer and Invertible Neural Network

MSCSCformer: Multiscale Convolutional Sparse Coding-Based Transformer for Pansharpening

A multimodal hyper-fusion transformer for remote sensing image classification

Effective Pan-Sharpening with Transformer and Invertible Neural Network

A Unified Generative Adversarial Network With Convolution and Transformer for Remote Sensing Image Fusion