Unsupervised Hyperspectral and Multispectral Image Fusion with Deep Spectral-Spatial Collaborative Constraint

Haoyang Yu,Zhixin Ling,Ke Zheng,Lianru Gao,Jiaxin Li,Jocelyn Chanussot
DOI: https://doi.org/10.1109/tgrs.2024.3472226
2024-01-01
Abstract:The most cost-effective way to obtain a high spatial resolution hyperspectral image (HrHSI) is to fuse low spatial resolution hyperspectral image (LrHSI) and corresponding high spatial resolution multispectral image (HrMSI). This paper proposes a generalizable unsupervised deep fusion method based on spectral-spatial collaborative constrain to address LrHSI and HrMSI fusion task. Firstly, in view of the limitations of the current spectral-spatial downsampled model, the group convolution enhancement (GCE) module is designed to eliminate radiometric difference between the images to be fused. Secondly, to enhance the model’s feature extraction ability, this paper introduces the design of the spatial, channel, and filter threedimensional attention factor dynamic convolutional kernel (SCFConv). In order to verify the proposed method, we compared and evaluated our method with traditional methods and unsupervised deep learning methods using both simulated and real on-board data, respectively. In the absence of HrHSI validation images in real scenarios, we evaluate the performance of different fusion models through classification results. The experimental results demonstrate the effectiveness of the proposed model and the practical value of the fusion results. (The on-board data produced by ours are available at https://drive.google.com/drive/folders/1JLCCB6ld5R49HDLN5SsMISx1d0fuqRjO).
What problem does this paper attempt to address?