Abstract:Fusion of images acquired using different sensors generates a single output with enhanced information for high-level visual perception applications. The transformer architecture has demonstrated its powerful ability to obtain important global contextual dependencies for multi-modal image fusion tasks. However, transformer-based image fusion methods face many critical issues, such as incurring huge computational burdens, limited ability to learn local features, and the difficulty of handling images of arbitrary sizes. To address the above limits, we proposed a novel Laplacian Pyramid Hybrid (LapH) network to combine the advantages of CNN and transformer architectures for multi-modal image fusion tasks. With the divide-and-conquer philosophy, we first build a light-weight CNN-based branch, performing effective extraction and fusion of texture/edge features via central difference convolutions, to process the high-resolution components with abundant details encoded in the lower pyramid levels of the Laplacian pyramid. Then, we design a transformer-based branch to process the low-resolution base components, learning long-range dependencies of global-contextual features without incurring extensive computational loads. Here, we design a multi-scale recurrent modulation mechanism to integrate the edge/texture features from the CNN branch as guidance to progressively refine the feature extraction and fusion on low-frequency components. Finally, we propose a new multi-scale spatial consistency loss term based on the neighbor contrast in source images, generating fused images with more natural and realistic appearances. Extensive experiments on two different multi-modal image fusion tasks verify the superiority of our method. The source codes are made publicly available at https://github.com/rgttadv/LapH .

Multi-modal Medical Image Fusion by Laplacian Pyramid and Adaptive Sparse Representation.

Image Fusion of CT and MR with Sparse Representation in NSST Domain

Medical Image Fusion Method by Using Laplacian Pyramid and Convolutional Sparse Representation

A Sum-Modified-laplacian and Sparse Representation Based Multimodal Medical Image Fusion in Laplacian Pyramid Domain

Medical Images Fusion by Using Weighted Least Squares Filter and Sparse Representation

Multi-Modal Image Fusion Via Deep Laplacian Pyramid Hybrid Network

Fusion of Medical Sensors Using Adaptive Cloud Model in Local Laplacian Pyramid Domain

Multi-Modal Image Fusion Via Sparse Representation and Multi-Scale Anisotropic Guided Measure

Recent Advances in Sparse Representation Based Medical Image Fusion

Multi-modal Medical Image Fusion Via Multi-Dictionary and Truncated Huber Filtering

An Image Fusion Method Based on Sparse Representation and Sum Modified-Laplacian in NSCT Domain

Multi-Modal Image Fusion Via Convolutional Morphological Component Analysis And Guided Filter

Medical Image Fusion Via Convolutional Sparsity Based Morphological Component Analysis

Multispectral and SAR Image Fusion Based on Laplacian Pyramid and Sparse Representation

Medical image fusion based on sparse representation and neighbor energy activity

Research of Multimodal Medical Image Fusion Based on Parameter-Adaptive Pulse-Coupled Neural Network and Convolutional Sparse Representation

Medical Image Fusion Based on Feature Extraction and Sparse Representation

MRI Image Fusion Based on Sparse Representation with Measurement of Patch-Based Multiple Salient Features

Medical Image Fusion Based on Sparse Representation and Guided Filtering

MRI and PET/SPECT fusion with pseudo spatial frequency in local laplacian filter domain

Multi-Dimensional Medical Image Fusion with Complex Sparse Representation