Abstract:multimodal image fusion involves tasks like pan-sharpening and depth super-resolution. Both tasks aim to generate high-resolution target images by fusing the complementary information from the texture-rich guidance and low-resolution target counterparts. They are inborn with reconstructing high-frequency information. Despite their inherent frequency domain connection, most existing methods only operate solely in the spatial domain and rarely explore the solutions in the frequency domain. This study addresses this limitation by proposing solutions in both the spatial and frequency domains. We devise a Spatial-Frequency Information Integration Network, abbreviated as SFINet for this purpose. The SFINet includes a core module tailored for image fusion. This module consists of three key components: a spatial-domain information branch, a frequency-domain information branch, and a dual-domain interaction. The spatial-domain information branch employs the spatial convolution-equipped invertible neural operators to integrate local information from different modalities in the spatial domain. Meanwhile, the frequency-domain information branch adopts a modality-aware deep Fourier transformation to capture the image-wide receptive field for exploring global contextual information. In addition, the dual-domain interaction facilitates information flow and the learning of complementary representations. We further present an improved version of SFINet, SFINet++, that enhances the representation of spatial information by replacing the basic convolution unit in the original spatial domain branch with the information-lossless invertible neural operator. We conduct extensive experiments to validate the effectiveness of the proposed networks and demonstrate their outstanding performance against state-of-the-art methods in two representative multimodal image fusion tasks: pan-sharpening and depth super-resolution. The source code is publicly available at https://github.com/manman1995/Awaresome-pansharpening.

Parallelizing Multi-sensor image fusion using CUDA with optimization

Detail Preserving Exposure Fusion for A Dual Sensor Camera

A Multi-Core Parallel Fusion Algorithm for Remote-Sensing Image

cuFSDAF: An Enhanced Flexible Spatiotemporal Data Fusion Algorithm Parallelized Using Graphics Processing Units

Multi-Modal Image Fusion Via Deep Laplacian Pyramid Hybrid Network

FusionGCN: Multi-focus image fusion using superpixel features generation GCN and pixel-level feature reconstruction CNN

Enabling Efficient Deep Convolutional Neural Network-based Sensor Fusion for Autonomous Driving

Multi-Modality Cascaded Fusion Technology for Autonomous Driving

An Efficient Implementation Method for Distributed Fusion in Sensor Networks Based on CPHD Filters

Fusion of multispectral satellite imagery using a cluster of graphics processing unit

Real-time and effective pan-sharpening for remote sensing using multi-scale fusion network

Towards Efficient Architecture and Algorithms for Sensor Fusion

Unified Cloud-Based Framework for Hyperspectral and Multispectral Image Fusion Incorporating Nonlocal Principles and Tensor Decomposition

A Novel Fusion Strategy and Convolutional Sparse Coding for Robot Multisource Image Fusion

Multi-Focus Image Fusion Algorithm in Sensor Networks

A General Spatial-Frequency Learning Framework for Multimodal Image Fusion

Improving the Performance of Image Fusion Based on Visual Saliency Weight Map Combined with CNN.

FusionArch: A Fusion-Based Accelerator for Point-Based Point Cloud Neural Networks

Pixel Level Image Fusion Methods and Fusion Performance Evaluation

Enabling Efficient Fast Convolution Algorithms on GPUs Via MegaKernels

An efficient parallel fusion structure of distilled and transformer-enhanced modules for lightweight image super-resolution