Abstract:multimodal image fusion involves tasks like pan-sharpening and depth super-resolution. Both tasks aim to generate high-resolution target images by fusing the complementary information from the texture-rich guidance and low-resolution target counterparts. They are inborn with reconstructing high-frequency information. Despite their inherent frequency domain connection, most existing methods only operate solely in the spatial domain and rarely explore the solutions in the frequency domain. This study addresses this limitation by proposing solutions in both the spatial and frequency domains. We devise a Spatial-Frequency Information Integration Network, abbreviated as SFINet for this purpose. The SFINet includes a core module tailored for image fusion. This module consists of three key components: a spatial-domain information branch, a frequency-domain information branch, and a dual-domain interaction. The spatial-domain information branch employs the spatial convolution-equipped invertible neural operators to integrate local information from different modalities in the spatial domain. Meanwhile, the frequency-domain information branch adopts a modality-aware deep Fourier transformation to capture the image-wide receptive field for exploring global contextual information. In addition, the dual-domain interaction facilitates information flow and the learning of complementary representations. We further present an improved version of SFINet, SFINet++, that enhances the representation of spatial information by replacing the basic convolution unit in the original spatial domain branch with the information-lossless invertible neural operator. We conduct extensive experiments to validate the effectiveness of the proposed networks and demonstrate their outstanding performance against state-of-the-art methods in two representative multimodal image fusion tasks: pan-sharpening and depth super-resolution. The source code is publicly available at https://github.com/manman1995/Awaresome-pansharpening.

A neuronet approach to information fusion.

Fusion of Low-Illuminance Visible and Near-Infrared Images Based on Convolutional Neural Networks

A hybrid NN-Bayesian architecture for information fusion

NLFNet: Non-Local Fusion Towards Generalized Multimodal Semantic Segmentation Across RGB-Depth, Polarization, and Thermal Images

A new method of image data fusion based on FNN

Fuzzy neural-network based state information fusion

CMCI: A Robust Multimodal Fusion Method for Spiking Neural Networks

Neural Dependency Coding inspired Multimodal Fusion

Information Fusion Based on Self-organizing Incremental Neural Network

A study on local sensor fusion of wireless sensor networks based on the neural network

PCNN Model Guided by Saliency Mechanism for Image Fusion in Transform Domain

Neural network image fusion with PCA preprocessing

NHFNET: A Non-Homogeneous Fusion Network for Multimodal Sentiment Analysis

A General Spatial-Frequency Learning Framework for Multimodal Image Fusion

Research and Application on NFE Model of Multi-Sensor Information Fusion

Nonlinear fusion is optimal for a wide class of multisensory tasks

CentralNet: a Multilayer Approach for Multimodal Fusion

CoCoNet: Coupled Contrastive Learning Network with Multi-level Feature Ensemble for Multi-modality Image Fusion

An artificial sensory neuron with visual-haptic fusion

Multi-Net Fusion: Exploring a Brain-Inspired Neural Network Model for Facial Expression Recognition

Research on NFE model of multi-sensor information fusion