Abstract:The fusion of infrared and visible images into a fused image that preserves the maximum information of both can share effective information from multiple modalities while maintaining the advantages of different modalities. Traditional deep fusion models mainly rely on convolution operations, which are relatively singular in the feature extraction process and prone to loss of feature information. Moreover, many networks also overlook the preservation of semantic information in obtaining visible image information. To address these shortcomings, we introduce TEPFusion, a new image fusion model that utilizes transformers and progressive fusion techniques. TEPFusion combines a Transformer-Convolutional module (TCM) to extract global and local key features, ensuring that the network pays attention to macro and micro details. In addition, a progressive channel fusion module (PCFM) is introduced for the fusion of multi-scale feature maps, achieving information integration at different channel scales. In order to further enrich the encoding features of the source image, we add dense connections to the encoder to ensure the preservation of valuable information throughout the entire encoding process. In addition, we design a loss function that includes gradient loss, strength loss, and semantic information loss, enabling our model to undergo unsupervised training. This method eliminates the need for manual design of complex fusion strategies required by traditional image fusion methods. Qualitative and quantitative experiments conducted on different public datasets have shown that our TEPFusion model performs better than other state-of-the-art methods, highlighting its generalization and high performance in infrared and visible image fusion. The ablation experiment prove the effectiveness of the TEPFusion architecture.

Infrared and Visible Image Fusion Based on Multiscale Adaptive Transformer

Fusion of infrared and visual images through multiscale hybrid unidirectional total variation

MFT: Multi-scale Fusion Transformer for Infrared and Visible Image Fusion

HitFusion: Infrared and Visible Image Fusion for High-Level Vision Tasks Using Transformer

Infrared and Visible Image Fusion Based on a Two-Stage Class Conditioned Auto-Encoder Network.

A Dual Cross Attention Transformer Network for Infrared and Visible Image Fusion

THFuse: An Infrared and Visible Image Fusion Network using Transformer and Hybrid Feature Extractor

Visible and Infrared Image Fusion Based on Attention and Multiscale Residuals

MFST: Multi-Modal Feature Self-Adaptive Transformer for Infrared and Visible Image Fusion

Multi-scale attention-based lightweight network with dilated convolutions for infrared and visible image fusion

MCFusion: infrared and visible image fusion based multiscale receptive field and cross-modal enhanced attention mechanism

TCCFusion: An Infrared and Visible Image Fusion Method based on Transformer and Cross Correlation

STFNet: Self-Supervised Transformer for Infrared and Visible Image Fusion

Infrared and visible image fusion based on visibility enhancement and hybrid multiscale decomposition

Fusion of Infrared and Visible Images Based on Three-Scale Decomposition and ResNet Feature Transfer

Multiscale Channel Attention Network for Infrared and Visible Image Fusion.

HDCTfusion: Hybrid Dual-Branch Network Based on CNN and Transformer for Infrared and Visible Image Fusion

Rethinking Cross-Attention for Infrared and Visible Image Fusion

HATF: Multi-Modal Feature Learning for Infrared and Visible Image Fusion via Hybrid Attention Transformer

A Cross-scale Iterative Attentional Adversarial Fusion Network for Infrared and Visible Images

TEPFusion: an End-to-End Infrared and Visible Image Fusion Network Based on Transformer and Progressive Channel Fusion