Abstract:Fusion of images acquired using different sensors generates a single output with enhanced information for high-level visual perception applications. The transformer architecture has demonstrated its powerful ability to obtain important global contextual dependencies for multi-modal image fusion tasks. However, transformer-based image fusion methods face many critical issues, such as incurring huge computational burdens, limited ability to learn local features, and the difficulty of handling images of arbitrary sizes. To address the above limits, we proposed a novel Laplacian Pyramid Hybrid (LapH) network to combine the advantages of CNN and transformer architectures for multi-modal image fusion tasks. With the divide-and-conquer philosophy, we first build a light-weight CNN-based branch, performing effective extraction and fusion of texture/edge features via central difference convolutions, to process the high-resolution components with abundant details encoded in the lower pyramid levels of the Laplacian pyramid. Then, we design a transformer-based branch to process the low-resolution base components, learning long-range dependencies of global-contextual features without incurring extensive computational loads. Here, we design a multi-scale recurrent modulation mechanism to integrate the edge/texture features from the CNN branch as guidance to progressively refine the feature extraction and fusion on low-frequency components. Finally, we propose a new multi-scale spatial consistency loss term based on the neighbor contrast in source images, generating fused images with more natural and realistic appearances. Extensive experiments on two different multi-modal image fusion tasks verify the superiority of our method. The source codes are made publicly available at https://github.com/rgttadv/LapH .

Image Fusion of Multiscale Contrast Pyramid-Based Vision Feature and Its Performance Evaluation

Fusion of infrared and visual images through multiscale hybrid unidirectional total variation

Multiscale Contrast Image Fusion Scheme with Performance Measures

Contrast-enhanced fusion of dual-band infrared image

Multi-Modal Image Fusion Via Deep Laplacian Pyramid Hybrid Network

Infrared And Visible Light Image Fusion Based On A Trous Contrast Pyramid And Nonsubsampled Directional Filter Bank

Fusion of Low-Quality Visible and Infrared Images Based on Multi-Level Latent Low-Rank Representation Joint with Retinex Enhancement and Multi-Visual Weight Information

Fusion of Infrared and Visible Images Via Multi-Layer Convolutional Sparse Representation

The Realistic Fusion of Multi-spectral Images

Performance assessment of image fusion

Pyramid Fully Convolutional Network for Hyperspectral and Multispectral Image Fusion.

A perceptual framework for infrared-visible image fusion based on multiscale structure decomposition and biological vision

Pixel Level Image Fusion Methods and Fusion Performance Evaluation

Multi-Level Image Fusion and Enhancement for Target Detection

Fusion algorithm of visible and infrared image based on anisotropic diffusion and image enhancement (capitalize only the first word in a title (or heading), the first word in a subtitle (or subheading), and any proper nouns)

A fast multi-resolution image fusion method based on visual features

Infrared and visible image fusion based on double fluid pyramids and multi-scale gradient residual block

Multispectral and SAR Image Fusion Based on Laplacian Pyramid and Sparse Representation

Comparisons of Several Pixel-Level Image Fusion Schemes for Infrared and Visible Light Images

Improving the Performance of Image Fusion Based on Visual Saliency Weight Map Combined with CNN.

A new image fusion performance metric based on visual information fidelity