Abstract:Fusion of images acquired using different sensors generates a single output with enhanced information for high-level visual perception applications. The transformer architecture has demonstrated its powerful ability to obtain important global contextual dependencies for multi-modal image fusion tasks. However, transformer-based image fusion methods face many critical issues, such as incurring huge computational burdens, limited ability to learn local features, and the difficulty of handling images of arbitrary sizes. To address the above limits, we proposed a novel Laplacian Pyramid Hybrid (LapH) network to combine the advantages of CNN and transformer architectures for multi-modal image fusion tasks. With the divide-and-conquer philosophy, we first build a light-weight CNN-based branch, performing effective extraction and fusion of texture/edge features via central difference convolutions, to process the high-resolution components with abundant details encoded in the lower pyramid levels of the Laplacian pyramid. Then, we design a transformer-based branch to process the low-resolution base components, learning long-range dependencies of global-contextual features without incurring extensive computational loads. Here, we design a multi-scale recurrent modulation mechanism to integrate the edge/texture features from the CNN branch as guidance to progressively refine the feature extraction and fusion on low-frequency components. Finally, we propose a new multi-scale spatial consistency loss term based on the neighbor contrast in source images, generating fused images with more natural and realistic appearances. Extensive experiments on two different multi-modal image fusion tasks verify the superiority of our method. The source codes are made publicly available at https://github.com/rgttadv/LapH .

Multi-Exposure Image Fusion via Multi-Scale and Context-Aware Feature Learning

Multi-Exposure Image Fusion via Deformable Self-Attention

Multi-exposure image fusion based on well-exposedness assessment

Multi-exposure image fusion via deep perceptual enhancement

An Ultra-High-definition Multi-Exposure Image Fusion Method Based on Multi-Scale Feature Extraction

Multi-Scale Exposure Fusion Via Content Adaptive Edge-Preserving Smoothing Pyramids.

Deep Guided Learning for Fast Multi-Exposure Image Fusion

TransMEF: A Transformer-Based Multi-Exposure Image Fusion Framework Using Self-Supervised Multi-Task Learning

Efficient Multi-exposure Image Fusion Via Filter-dominated Fusion and Gradient-driven Unsupervised Learning.

Multiscale Feature Interactive Network for Multifocus Image Fusion

Ref-MEF: Reference-Guided Flexible Gated Image Reconstruction Network for Multi-Exposure Image Fusion

Coarse-to-fine multi-scale attention-guided network for multi-exposure image fusion

A Dual Domain Multi-exposure Image Fusion Network based on the Spatial-Frequency Integration

Multi Exposure Fusion for High Dynamic Range Imaging Via Multi-Channel Gradient Tensor

Rethinking Multi-Exposure Image Fusion with Extreme and Diverse Exposure Levels: A Robust Framework Based on Fourier Transform and Contrastive Learning.

An enhanced multi-scale weight assignment strategy of two-exposure fusion

Multi-Modal Image Fusion Via Deep Laplacian Pyramid Hybrid Network

Multiexposure Fusion Based on Multiscale Edge Information Measurement and Adaptive Pyramid Optimization

Multi-focus image fusion with deep residual learning and focus property detection

GALFusion: Multi-Exposure Image Fusion via a Global–Local Aggregation Learning Network