Abstract:Fusion of images acquired using different sensors generates a single output with enhanced information for high-level visual perception applications. The transformer architecture has demonstrated its powerful ability to obtain important global contextual dependencies for multi-modal image fusion tasks. However, transformer-based image fusion methods face many critical issues, such as incurring huge computational burdens, limited ability to learn local features, and the difficulty of handling images of arbitrary sizes. To address the above limits, we proposed a novel Laplacian Pyramid Hybrid (LapH) network to combine the advantages of CNN and transformer architectures for multi-modal image fusion tasks. With the divide-and-conquer philosophy, we first build a light-weight CNN-based branch, performing effective extraction and fusion of texture/edge features via central difference convolutions, to process the high-resolution components with abundant details encoded in the lower pyramid levels of the Laplacian pyramid. Then, we design a transformer-based branch to process the low-resolution base components, learning long-range dependencies of global-contextual features without incurring extensive computational loads. Here, we design a multi-scale recurrent modulation mechanism to integrate the edge/texture features from the CNN branch as guidance to progressively refine the feature extraction and fusion on low-frequency components. Finally, we propose a new multi-scale spatial consistency loss term based on the neighbor contrast in source images, generating fused images with more natural and realistic appearances. Extensive experiments on two different multi-modal image fusion tasks verify the superiority of our method. The source codes are made publicly available at https://github.com/rgttadv/LapH .

Multilevel Features Fusion In Deep Convolutional Neural Networks

Fusion of Low-Illuminance Visible and Near-Infrared Images Based on Convolutional Neural Networks

Aerial Scene Classification Via Multilevel Fusion Based on Deep Convolutional Neural Networks.

Improving the Performance of Convolutional Neural Networks by Fusing Low-Level Features with Different Scales in the Preceding Stage

A Late Fusion Approach for Harnessing Multi-Cnn Model High-Level Features

Multilayer Feature Descriptors Fusion CNN Models for Fine-Grained Visual Recognition

Multi-Modal Image Fusion Via Deep Laplacian Pyramid Hybrid Network

Very High Resolution Images Classification by Fusing Deep Convolutional Neural Networks

Fusion that matters: convolutional fusion networks for visual recognition

MFFN: image super-resolution via multi-level features fusion network

Multilevel feature fusion dilated convolutional network for semantic segmentation

Scene Classification Based on Two-Stage Deep Feature Fusion

On Fusing the Latent Deep CNN Feature for Image Classification.

Self-Adaption Multi-classifier Fusion Networks for Image Recognition

AMFF-net: Adaptive Multi-Modal Feature Fusion Network for Image Classification

Reuse of Mid-Level Feature in Deep Convolutional Neural Network

Multiple Layers Global Average Pooling Fusion

Fully Dense Multiscale Fusion Network for Hyperspectral Image Classification

Multi-Focus Image Fusion with a Natural Enhancement Via a Joint Multi-Level Deeply Supervised Convolutional Neural Network

Hyperspectral Remote Sensing Images Deep Feature Extraction Based on Mixed Feature and Convolutional Neural Networks

On the Exploration of Convolutional Fusion Networks for Visual Recognition.