Abstract:multimodal image fusion involves tasks like pan-sharpening and depth super-resolution. Both tasks aim to generate high-resolution target images by fusing the complementary information from the texture-rich guidance and low-resolution target counterparts. They are inborn with reconstructing high-frequency information. Despite their inherent frequency domain connection, most existing methods only operate solely in the spatial domain and rarely explore the solutions in the frequency domain. This study addresses this limitation by proposing solutions in both the spatial and frequency domains. We devise a Spatial-Frequency Information Integration Network, abbreviated as SFINet for this purpose. The SFINet includes a core module tailored for image fusion. This module consists of three key components: a spatial-domain information branch, a frequency-domain information branch, and a dual-domain interaction. The spatial-domain information branch employs the spatial convolution-equipped invertible neural operators to integrate local information from different modalities in the spatial domain. Meanwhile, the frequency-domain information branch adopts a modality-aware deep Fourier transformation to capture the image-wide receptive field for exploring global contextual information. In addition, the dual-domain interaction facilitates information flow and the learning of complementary representations. We further present an improved version of SFINet, SFINet++, that enhances the representation of spatial information by replacing the basic convolution unit in the original spatial domain branch with the information-lossless invertible neural operator. We conduct extensive experiments to validate the effectiveness of the proposed networks and demonstrate their outstanding performance against state-of-the-art methods in two representative multimodal image fusion tasks: pan-sharpening and depth super-resolution. The source code is publicly available at https://github.com/manman1995/Awaresome-pansharpening.

Multiscale Feature Interactive Network for Multifocus Image Fusion

StackMFF: End-to-end Multi-Focus Image Stack Fusion Network

Multi-Scale Cross-Attention Fusion Network Based on Image Super-Resolution

Multi-focused image fusion algorithm based on multi-scale hybrid attention residual network

MM-Net: A MixFormer-Based Multi-Scale Network for Anatomical and Functional Image Fusion

Multi-Focus Image Fusion Using U-Shaped Networks with a Hybrid Objective

Multi-focus image fusion with deep residual learning and focus property detection

Focus Affinity Perception and Super-Resolution Embedding for Multifocus Image Fusion

Multi-focus image fusion with a deep convolutional neural network

Multi-Exposure Image Fusion via Multi-Scale and Context-Aware Feature Learning

A Novel Multiscale Transform Decomposition Based Multi-Focus Image Fusion Framework

New Insights into Multi-focus Image Fusion: A Fusion Method Based on Multi-dictionary Linear Sparse Representation and Region Fusion Model

A Self-Supervised Residual Feature Learning Model for Multifocus Image Fusion

A multi-channel neural network model for multi-focus image fusion

MAFusion: Multiscale Attention Network for Infrared and Visible Image Fusion

SADFusion: A multi-scale infrared and visible image fusion method based on salient-aware and domain-specific

Multi-focus Image Fusion Using Fully Convolutional Two-stream Network for Visual Sensors.

Fusion-UDCGAN: Multifocus Image Fusion via a U-Type Densely Connected Generation Adversarial Network

A General Spatial-Frequency Learning Framework for Multimodal Image Fusion

Exploit the Best of Both End-to-End and Map-Based Methods for Multi-Focus Image Fusion

Multi-scale unsupervised network for infrared and visible image fusion based on joint attention mechanism