Abstract:multimodal image fusion involves tasks like pan-sharpening and depth super-resolution. Both tasks aim to generate high-resolution target images by fusing the complementary information from the texture-rich guidance and low-resolution target counterparts. They are inborn with reconstructing high-frequency information. Despite their inherent frequency domain connection, most existing methods only operate solely in the spatial domain and rarely explore the solutions in the frequency domain. This study addresses this limitation by proposing solutions in both the spatial and frequency domains. We devise a Spatial-Frequency Information Integration Network, abbreviated as SFINet for this purpose. The SFINet includes a core module tailored for image fusion. This module consists of three key components: a spatial-domain information branch, a frequency-domain information branch, and a dual-domain interaction. The spatial-domain information branch employs the spatial convolution-equipped invertible neural operators to integrate local information from different modalities in the spatial domain. Meanwhile, the frequency-domain information branch adopts a modality-aware deep Fourier transformation to capture the image-wide receptive field for exploring global contextual information. In addition, the dual-domain interaction facilitates information flow and the learning of complementary representations. We further present an improved version of SFINet, SFINet++, that enhances the representation of spatial information by replacing the basic convolution unit in the original spatial domain branch with the information-lossless invertible neural operator. We conduct extensive experiments to validate the effectiveness of the proposed networks and demonstrate their outstanding performance against state-of-the-art methods in two representative multimodal image fusion tasks: pan-sharpening and depth super-resolution. The source code is publicly available at https://github.com/manman1995/Awaresome-pansharpening.

A General Spatial-Frequency Learning Framework for Multimodal Image Fusion

Efficient Perceiving Local Details Via Adaptive Spatial-Frequency Information Integration for Multi-focus Image Fusion

Frequency Integration and Spatial Compensation Network for Infrared and Visible Image Fusion

SFCFusion: Spatial–Frequency Collaborative Infrared and Visible Image Fusion

AFDFusion: an Adaptive Frequency Decoupling Fusion Network for Multi-Modality Image

SpFusionNet: deep learning-driven brain image fusion with spatial frequency analysis

Revisiting Spatial-Frequency Information Integration from a Hierarchical Perspective for Panchromatic and Multi-Spectral Image Fusion

A General Image Fusion Framework Using Multi-Task Semi-Supervised Learning

Bridging the Gap between Multi-focus and Multi-modal: A Focused Integration Framework for Multi-modal Image Fusion

A Unified Image Fusion Framework with Flexible Bilevel Paradigm Integration

Where Elegance Meets Precision: Towards a Compact, Automatic, and Flexible Framework for Multi-modality Image Fusion and Applications

Multiscale Feature Interactive Network for Multifocus Image Fusion

CHFusion: A Cross-modality High-resolution Representation Framework for Infrared and Visible Image Fusion

UNIFusion: A Lightweight Unified Image Fusion Network

SADFusion: A multi-scale infrared and visible image fusion method based on salient-aware and domain-specific

Focus Affinity Perception and Super-Resolution Embedding for Multifocus Image Fusion

SGFusion: A Saliency Guided Deep-Learning Framework for Pixel-Level Image Fusion

SFINet: A semantic feature interactive learning network for full-time infrared and visible image fusion

SuperFusion: A Versatile Image Registration and Fusion Network with Semantic Awareness

SFDFusion: An Efficient Spatial-Frequency Domain Fusion Network for Infrared and Visible Image Fusion

A Multi-Modal Image Fusion Framework Based on Guided Filter and Sparse Representation