DDRF: Dual-branch decomposition and reconstruction architecture for infrared and visible image fusion

Lei Zhang,Qiming Zhou,Mingliang Tang,Xin Ding,Chengwei Yang,Chuyuan Wei,Zhimiao Zhou

DOI: https://doi.org/10.1016/j.optlastec.2024.111991

IF: 4.939

2024-11-08

Optics & Laser Technology

Abstract:Infrared and visible image fusion is an important image enhancement technique. It aims to combine information from different modalities to produce high-quality fusion images with prominent targets and rich textures. However, current image fusion methods cannot adequately extract meaningful features from modalities. So, this paper proposes a Dual-Branch Decomposition and Reconstruction Fusion (DDRF) architecture. Initially, DDRF uses residual XCiT blocks to extract shallow features from modalities. We then introduce a dual-branch Transformer-CNN feature extractor with lightweight, high-quality Base Feature Encoder Module (BFEM) and Detail Feature Encoder Module (DFEM). BFEM utilizes global attention to process low-frequency base features, while DFEM focuses on extracting high-frequency detail features. Furthermore, the fused image is generated through feature fusion and reconstruction. The combination of BFEM and DFEM not only improves the accuracy of feature extraction, but also optimizes information retention during the fusion process. Extensive experiments demonstrate that DDRF achieves excellent results in infrared and visible image fusion, especially in medical image fusion, and enhances downstream infrared–visible object detection performance.

optics,physics, applied