Abstract:multimodal image fusion involves tasks like pan-sharpening and depth super-resolution. Both tasks aim to generate high-resolution target images by fusing the complementary information from the texture-rich guidance and low-resolution target counterparts. They are inborn with reconstructing high-frequency information. Despite their inherent frequency domain connection, most existing methods only operate solely in the spatial domain and rarely explore the solutions in the frequency domain. This study addresses this limitation by proposing solutions in both the spatial and frequency domains. We devise a Spatial-Frequency Information Integration Network, abbreviated as SFINet for this purpose. The SFINet includes a core module tailored for image fusion. This module consists of three key components: a spatial-domain information branch, a frequency-domain information branch, and a dual-domain interaction. The spatial-domain information branch employs the spatial convolution-equipped invertible neural operators to integrate local information from different modalities in the spatial domain. Meanwhile, the frequency-domain information branch adopts a modality-aware deep Fourier transformation to capture the image-wide receptive field for exploring global contextual information. In addition, the dual-domain interaction facilitates information flow and the learning of complementary representations. We further present an improved version of SFINet, SFINet++, that enhances the representation of spatial information by replacing the basic convolution unit in the original spatial domain branch with the information-lossless invertible neural operator. We conduct extensive experiments to validate the effectiveness of the proposed networks and demonstrate their outstanding performance against state-of-the-art methods in two representative multimodal image fusion tasks: pan-sharpening and depth super-resolution. The source code is publicly available at https://github.com/manman1995/Awaresome-pansharpening.

SpFusionNet: deep learning-driven brain image fusion with spatial frequency analysis

A General Spatial-Frequency Learning Framework for Multimodal Image Fusion

Multi-focus image fusion with a deep convolutional neural network

Adaptive Multimodal Image Fusion with a Deep Pyramidal Residual Learning Network

The Multi-Focus-Image-Fusion Method Based on Convolutional Neural Network and Sparse Representation.

Efficient Perceiving Local Details Via Adaptive Spatial-Frequency Information Integration for Multi-focus Image Fusion

Brain CT and MRI medical image fusion using convolutional neural networks and a dual-channel spiking cortical model

Multi-scale Convolutional Neural Network for Multi-Focus Image Fusion.

A novel approach for multi-focus image fusion based on SF-PAPCNN and ISML in NSST domain

A fuzzy convolutional neural network for enhancing multi-focus image fusion

MM-Net: A MixFormer-Based Multi-Scale Network for Anatomical and Functional Image Fusion

SFCFusion: Spatial–Frequency Collaborative Infrared and Visible Image Fusion

Multimodal Magnetic Resonance Image and Electroencephalogram Constrained Fusion Algorithm Using Deep Learning

SESF-Fuse: an unsupervised deep model for multi-focus image fusion

Multi-modal Medical Neurological Image Fusion using Wavelet Pooled Edge Preserving Autoencoder

M4FNet: Multimodal Medical Image Fusion Network Via Multi-Receptive-field and Multi-Scale Feature Integration.

Multi-Focus Image Fusion with a Natural Enhancement Via a Joint Multi-Level Deeply Supervised Convolutional Neural Network

Adaptive spatial and frequency experts fusion network for medical image fusion

Multi-focus Image Fusion Using Adaptive Region and SCM Based on NSST Domain

Better and Faster Deep Image Fusion with Spatial Frequency

Multimodal MRI Volumetric Data Fusion With Convolutional Neural Networks