Abstract:Fusion of images acquired using different sensors generates a single output with enhanced information for high-level visual perception applications. The transformer architecture has demonstrated its powerful ability to obtain important global contextual dependencies for multi-modal image fusion tasks. However, transformer-based image fusion methods face many critical issues, such as incurring huge computational burdens, limited ability to learn local features, and the difficulty of handling images of arbitrary sizes. To address the above limits, we proposed a novel Laplacian Pyramid Hybrid (LapH) network to combine the advantages of CNN and transformer architectures for multi-modal image fusion tasks. With the divide-and-conquer philosophy, we first build a light-weight CNN-based branch, performing effective extraction and fusion of texture/edge features via central difference convolutions, to process the high-resolution components with abundant details encoded in the lower pyramid levels of the Laplacian pyramid. Then, we design a transformer-based branch to process the low-resolution base components, learning long-range dependencies of global-contextual features without incurring extensive computational loads. Here, we design a multi-scale recurrent modulation mechanism to integrate the edge/texture features from the CNN branch as guidance to progressively refine the feature extraction and fusion on low-frequency components. Finally, we propose a new multi-scale spatial consistency loss term based on the neighbor contrast in source images, generating fused images with more natural and realistic appearances. Extensive experiments on two different multi-modal image fusion tasks verify the superiority of our method. The source codes are made publicly available at https://github.com/rgttadv/LapH .

Pyramid Pooling Dense Convolutional Neural Network for Multi-focus Image Fusion

Multi-Modal Image Fusion Via Deep Laplacian Pyramid Hybrid Network

Multi-focus image fusion with a deep convolutional neural network

Multi-focus Image Fusion with Convolutional Neural Network Based on Dempster-Shafer Theory

Multi-focus Image Fusion Based on Fully Convolutional Networks.

CCSR-Net: Unfolding Coupled Convolutional Sparse Representation for Multi-focus Image Fusion.

Multi-focus Image Fusion Using Fully Convolutional Two-stream Network for Visual Sensors.

Multi-scale Convolutional Neural Network for Multi-Focus Image Fusion.

Pyramid Fully Convolutional Network for Hyperspectral and Multispectral Image Fusion.

Multi-focus image fusion with deep residual learning and focus property detection

Multi-focus Image Fusion Using Deep Support Value Convolutional Neural Network

Multi-focus image fusion with the all convolutional neural network

Multi-focused image fusion algorithm based on multi-scale hybrid attention residual network

Image Segmentation-Based Multi-Focus Image Fusion Through Multi-Scale Convolutional Neural Network.

Fine-grained multi-focus image fusion based on edge features

DDFusion: an Efficient Multi-Exposure Fusion Network with Dense Pyramidal Convolution and De-Correlation Fusion

A multi-focus image fusion network with local-global joint attention module

Multi-Focus Image Fusion with a Natural Enhancement Via a Joint Multi-Level Deeply Supervised Convolutional Neural Network

Multi-focus Image Fusion Using Structure-Guided Flow.

Multi-focus image fusion by progressive pixel extraction

Multifocus Image Fusion Using Convolutional Neural Network