Abstract:Fusion of images acquired using different sensors generates a single output with enhanced information for high-level visual perception applications. The transformer architecture has demonstrated its powerful ability to obtain important global contextual dependencies for multi-modal image fusion tasks. However, transformer-based image fusion methods face many critical issues, such as incurring huge computational burdens, limited ability to learn local features, and the difficulty of handling images of arbitrary sizes. To address the above limits, we proposed a novel Laplacian Pyramid Hybrid (LapH) network to combine the advantages of CNN and transformer architectures for multi-modal image fusion tasks. With the divide-and-conquer philosophy, we first build a light-weight CNN-based branch, performing effective extraction and fusion of texture/edge features via central difference convolutions, to process the high-resolution components with abundant details encoded in the lower pyramid levels of the Laplacian pyramid. Then, we design a transformer-based branch to process the low-resolution base components, learning long-range dependencies of global-contextual features without incurring extensive computational loads. Here, we design a multi-scale recurrent modulation mechanism to integrate the edge/texture features from the CNN branch as guidance to progressively refine the feature extraction and fusion on low-frequency components. Finally, we propose a new multi-scale spatial consistency loss term based on the neighbor contrast in source images, generating fused images with more natural and realistic appearances. Extensive experiments on two different multi-modal image fusion tasks verify the superiority of our method. The source codes are made publicly available at https://github.com/rgttadv/LapH .

PT-Net: Pyramid Transformer Network for Feature Matching Learning

Pyramid Point Cloud Transformer for Large-Scale Place Recognition.

P2T: Pyramid Pooling Transformer for Scene Understanding

OVPT: Optimal Viewset Pooling Transformer for 3D Object Recognition.

InvPT: Inverted Pyramid Multi-task Transformer for Dense Scene Understanding

Multi-Modal Image Fusion Via Deep Laplacian Pyramid Hybrid Network

Multidirection and Multiscale Pyramid in Transformer for Video-Based Pedestrian Retrieval

Pyramid Learnable Tokens for 3D LiDAR Place Recognition.

DCPNet: A Densely Connected Pyramid Network for Monocular Depth Estimation

PTT: Point-Track-Transformer Module for 3D Single Object Tracking in Point Clouds

Multi-direction and Multi-scale Pyramid in Transformer for Video-based Pedestrian Retrieval

3DCTN: 3D Convolution-Transformer Network for Point Cloud Classification

Pyramid Feature Attention Network for Monocular Depth Prediction

Transformer-guided Feature Pyramid Network for Multi-View Stereo

Cascaded Pyramid Network for Multi-Person Pose Estimation

Complementary Feature Pyramid Network for Object Detection

Cross Pyramid Transformer makes U-net stronger in medical image segmentation

Multi-scale Cross-form Pyramid Network for Stereo Matching

Pyramid R-CNN: Towards Better Performance and Adaptability for 3D Object Detection

Pyramid Vision Transformer: A Versatile Backbone for Dense Prediction without Convolutions

UFO-Net: A Linear Attention-Based Network for Point Cloud Classification