Unsupervised Hyperspectral and Multispectral Image Blind Fusion Based on Deep Tucker Decomposition Network with Spatial-Spectral Manifold Learning

He Wang,Yang Xu,Zebin Wu,Zhihui Wei

2024-09-19

Abstract:Hyperspectral and multispectral image fusion aims to generate high spectral and spatial resolution hyperspectral images (HR-HSI) by fusing high-resolution multispectral images (HR-MSI) and low-resolution hyperspectral images (LR-HSI). However, existing fusion methods encounter challenges such as unknown degradation parameters, incomplete exploitation of the correlation between high-dimensional structures and deep image features. To overcome these issues, in this article, an unsupervised blind fusion method for hyperspectral and multispectral images based on Tucker decomposition and spatial spectral manifold learning (DTDNML) is proposed. We design a novel deep Tucker decomposition network that maps LR-HSI and HR-MSI into a consistent feature space, achieving reconstruction through decoders with shared parameter. To better exploit and fuse spatial-spectral features in the data, we design a core tensor fusion network that incorporates a spatial spectral attention mechanism for aligning and fusing features at different scales. Furthermore, to enhance the capacity in capturing global information, a Laplacian-based spatial-spectral manifold constraints is introduced in shared-decoders. Sufficient experiments have validated that this method enhances the accuracy and efficiency of hyperspectral and multispectral fusion on different remote sensing datasets. The source code is available at <a class="link-external link-https" href="https://github.com/Shawn-H-Wang/DTDNML" rel="external noopener nofollow">this https URL</a>.

Computer Vision and Pattern Recognition,Image and Video Processing

What problem does this paper attempt to address?

The paper attempts to address the problem of how to fuse low-resolution hyperspectral images (LR-HSI) with high-resolution multispectral images (HR-MSI) to generate hyperspectral images (HR-HSI) with high spatial and spectral resolution without the need for known degradation parameters. Existing fusion methods face several challenges when dealing with this problem, such as unknown degradation parameters and the failure to fully utilize the correlation between high-dimensional structures and deep image features. To this end, this paper proposes an unsupervised blind fusion method based on a deep Tucker decomposition network and spatial-spectral manifold learning (DTDNML), aiming to overcome the above challenges and improve the accuracy and efficiency of fusion.

Specifically, the method designs a novel deep Tucker decomposition network to map LR-HSI and HR-MSI into a consistent feature space and achieve reconstruction through a decoder with shared parameters. To better utilize and fuse the spatial-spectral features in the data, a core tensor fusion network is designed, which combines a spatial-spectral attention mechanism to align and fuse features at different scales. Additionally, to enhance the ability to capture global information, a Laplacian-based spatial-spectral manifold constraint is introduced in the shared decoder. Experimental results show that this method improves the accuracy and efficiency of hyperspectral and multispectral image fusion on different remote sensing datasets.