Abstract:Multimodal image fusion (MMIF) can provide more comprehensive scene characteristics by synthesizing a single image from multi-sensor images of the same scene, which works out the limitation of single-type hardware. To handle MMIF tasks, current deep learning (DL)-based methods usually use convolutional neural networks (CNNs) or combine transformer to extract local and global contextual information of source images. However, none of the existing works fully explores contextual information both across modalities and within single modalities, leading to limited fusion results. To this end, we propose a new MMIF method via locally enhanced global learning, termed as LeGFusion. Specifically, the network of LeGFusion is devised based on locally enhanced transformer block (LETB), which can capture long-range dependencies benefiting from nonoverlapping window-based self-attention while capturing useful local context with the utilization of the convolution operator into transformer. On one hand, several LETBs are deployed to extract global contexts from each modality while emphasizing its local information. On the other hand, the fusion module that also consists of LETBs is designed to integrate multimodal features by perceiving cross-modal local and global interactions. Powered by these intramodal and intermodal contextual information exploration, the proposed LeGFusion enjoys a high capability in capturing significant complementary information for image fusion. Extensive experiments are conducted on two types of MMIF tasks, including infrared-visible image fusion (IVF) and medical image fusion. The qualitative and quantitative evaluation results demonstrate the superiority of our LeGFusion over state-of-the-art methods. Furthermore, we validate the generalization ability of LeGFusion without fine-tuning and achieve fantastic results.

MMFusion: Combining Image Forensic Filters for Visual Manipulation Detection and Localization

Fusion Transformer with Object Mask Guidance for Image Forgery Analysis

DMFF-Net: Double-stream multilevel feature fusion network for image forgery localization

Image Manipulation Localization Using Spatial–Channel Fusion Excitation and Fine-Grained Feature Enhancement

Harmonizing Image Forgery Detection & Localization: Fusion of Complementary Approaches

MGFuse: An Infrared and Visible Image Fusion Algorithm Based on Multiscale Decomposition Optimization and Gradient-Weighted Local Energy

Bridging the Gap between Multi-focus and Multi-modal: A Focused Integration Framework for Multi-modal Image Fusion

Image Manipulation Localization Using Multi-Scale Feature Fusion and Adaptive Edge Supervision

Image splicing region localization with adaptive multi-feature filtration

MMDRFuse: Distilled Mini-Model with Dynamic Refresh for Multi-Modality Image Fusion

LeGFusion: Locally Enhanced Global Learning for Multimodal Image Fusion

mmFUSION: Multimodal Fusion for 3D Objects Detection

LeGFusion: Locally-enhanced Global Learning for Multi-Modal Image Fusion

Image Forgery Detection and Localization via a Reliability Fusion Map

Multi-modal Image Fusion with the Hybrid ℓ0ℓ1 Layer Decomposing and Multi-Directional Filter Banks

Mutli-focus image fusion based on guided filter and image matting network

Multi-Focus Image Fusion Using U-Shaped Networks with a Hybrid Objective

New Insights into Multi-focus Image Fusion: A Fusion Method Based on Multi-dictionary Linear Sparse Representation and Region Fusion Model

SAMF: Small-Area-Aware Multi-focus Image Fusion for Object Detection

Rethinking Normalization Strategies and Convolutional Kernels for Multimodal Image Fusion

Mutual-Guided Dynamic Network for Image Fusion