Fourier-enhanced Implicit Neural Fusion Network for Multispectral and Hyperspectral Image Fusion

Yu-Jie Liang,Zihan Cao,Liang-Jian Deng,Xiao Wu

2024-04-24

Abstract:Recently, implicit neural representations (INR) have made significant strides in various vision-related domains, providing a novel solution for Multispectral and Hyperspectral Image Fusion (MHIF) tasks. However, INR is prone to losing high-frequency information and is confined to the lack of global perceptual capabilities. To address these issues, this paper introduces a Fourier-enhanced Implicit Neural Fusion Network (FeINFN) specifically designed for MHIF task, targeting the following phenomena: The Fourier amplitudes of the HR-HSI latent code and LR-HSI are remarkably similar; however, their phases exhibit different patterns. In FeINFN, we innovatively propose a spatial and frequency implicit fusion function (Spa-Fre IFF), helping INR capture high-frequency information and expanding the receptive field. Besides, a new decoder employing a complex Gabor wavelet activation function, called Spatial-Frequency Interactive Decoder (SFID), is invented to enhance the interaction of INR features. Especially, we further theoretically prove that the Gabor wavelet activation possesses a time-frequency tightness property that favors learning the optimal bandwidths in the decoder. Experiments on two benchmark MHIF datasets verify the state-of-the-art (SOTA) performance of the proposed method, both visually and quantitatively. Also, ablation studies demonstrate the mentioned contributions. The code will be available on Anonymous GitHub (https://anonymous.4open.science/r/FeINFN-15C9/) after possible acceptance.

Computer Vision and Pattern Recognition

What problem does this paper attempt to address?

This paper attempts to solve two main problems in the multi - spectral and hyperspectral image fusion (MHIF) task:

1. **Loss of high - frequency information**: Implicit neural representation (INR) is prone to lose high - frequency information when processing images, which results in the fused image lacking in details and being unable to fully preserve the fine structure of the original image.
2. **Insufficient global perception ability**: Traditional INR methods are limited by local operations and lack effective utilization of global information, which restricts their performance in complex scenes.

To address these problems, the authors propose a Fourier - enhanced implicit neural fusion network (FeINFN). This network improves the performance of INR in the MHIF task by introducing the spatial and frequency implicit fusion function (Spa - Fre IFF) and a new decoder - the spatial - frequency interaction decoder (SFID).

### Main contributions

1. **Innovative fusion framework**: A new fusion framework is proposed. Based on INR, this framework simultaneously performs information extraction and fusion in the spatial and frequency domains, effectively enhancing the representation ability of high - frequency information and expanding the receptive field.
2. **New decoder design**: A decoder using the complex - valued Gabor wavelet activation function is designed. This activation function has time - frequency compactness, which helps the decoder learn the optimal bandwidth and thus enhances the feature interaction.
3. **Excellent experimental results**: Experiments were carried out on two widely - used hyperspectral datasets (CAVE and Harvard). The results show that the proposed FeINFN achieves state - of - the - art performance on multiple evaluation metrics, including PSNR, SAM, ERGAS, and SSIM.

### Method overview

1. **INR encoder**: Spectral features and spatial features are respectively extracted from the low - resolution hyperspectral image (LR - HSI) and the high - resolution multi - spectral image (HR - MSI) through two encoders.
2. **Spatial and frequency implicit fusion function (Spa - Fre IFF)**: At the queried high - resolution coordinates, the spatial feature vector and the frequency feature vector are estimated, and the features are fused through the implicit fusion function.
3. **Spatial - frequency interaction decoder (SFID)**: The spatial feature map and the frequency - domain feature map are seamlessly integrated to generate the final fused image. The decoder uses the complex - valued Gabor wavelet activation function and has good time - frequency compactness.

### Experimental results

- **CAVE dataset**: Under a ×4 scaling factor, FeINFN achieves the optimal performance on multiple metrics such as PSNR, SAM, ERGAS, and SSIM, significantly outperforming existing methods.
- **Harvard dataset**: Also under a ×4 scaling factor, FeINFN performs excellently on PSNR, ERGAS, and SSIM metrics, and is only slightly inferior to 3DT - Net and BDT on the SAM metric.

### Conclusion

This paper effectively solves the problems of high - frequency information loss and insufficient global perception ability in traditional INR methods for multi - spectral and hyperspectral image fusion tasks by introducing the Fourier - enhanced implicit neural fusion network (FeINFN). The experimental results verify the superior performance of this method on multiple datasets and show its potential in high - resolution hyperspectral image generation.