Abstract:Infrared and visible image fusion aims to generate a single fused image that not only contains rich texture details and salient objects, but also facilitates downstream tasks. However, existing works mainly focus on learning different modality-specific or shared features, and ignore the importance of modeling cross-modality features. To address these challenges, we propose Dual-branch Progressive learning for infrared and visible image fusion with a complementary self-Attention and Convolution (DPACFuse) network. On the one hand, we propose Cross-Modality Feature Extraction (CMEF) to enhance information interaction and the extraction of common features across modalities. In addition, we introduce a high-frequency gradient convolution operation to extract fine-grained information and suppress high-frequency information loss. On the other hand, to alleviate the CNN issues of insufficient global information extraction and computation overheads of self-attention, we introduce the ACmix, which can fully extract local and global information in the source image with a smaller computational overhead than pure convolution or pure self-attention. Extensive experiments demonstrated that the fused images generated by DPACFuse not only contain rich texture information, but can also effectively highlight salient objects. Additionally, our method achieved approximately 3% improvement over the state-of-the-art methods in MI, Qabf, SF, and AG evaluation indicators. More importantly, our fused images enhanced object detection and semantic segmentation by approximately 10%, compared to using infrared and visible images separately.

DeepFuse Neural Networks

ExFuse: Enhancing Feature Fusion for Semantic Segmentation

FAFuse: A Four-Axis Fusion framework of CNN and transformer for medical image segmentation

Aerial Scene Classification Via Multilevel Fusion Based on Deep Convolutional Neural Networks.

ObjectFusion: an Object Detection and Segmentation Framework with RGB-D SLAM and Convolutional Neural Networks

THFuse: An Infrared and Visible Image Fusion Network using Transformer and Hybrid Feature Extractor

MBHFuse: A Multi- Branch Heterogeneous Global and Local Infrared and Visible Image Fusion with Differential Convolutional Amplification Features

A Late Fusion Approach for Harnessing Multi-Cnn Model High-Level Features

CoTrFuse: a novel framework by fusing CNN and transformer for medical image segmentation

Deep Fusion Net for Multi-atlas Segmentation: Application to Cardiac MR Images

TransFuse: Fusing Transformers and CNNs for Medical Image Segmentation

DPACFuse: Dual-Branch Progressive Learning for Infrared and Visible Image Fusion with Complementary Self-Attention and Convolution

FusNet: A Deep Information Fusion Network for Medical Image Segmentation

DenseFuse: A Fusion Approach to Infrared and Visible Images

Mask-R-FCN: A Deep Fusion Network for Semantic Segmentation.

Exploring Multi-scale Deep Feature Fusion for Object Detection.

Feature Fusion Use Unsupervised Prior Knowledge to Let Small Object Represent

AMFuse: Add–Multiply-Based Cross-Modal Fusion Network for Multi-Spectral Semantic Segmentation

Fused DNN: A deep neural network fusion approach to fast and robust pedestrian detection

CMFuse: Cross-Modal Features Mixing Via Convolution and MLP for Infrared and Visible Image Fusion

FuseNet: 3D Object Detection Network with Fused Information for Lidar Point Clouds.