MBHFuse: A Multi- Branch Heterogeneous Global and Local Infrared and Visible Image Fusion with Differential Convolutional Amplification Features

Yichen Sun,Mingli Dong,Mingxin Yu,Lianqing Zhu

DOI: https://doi.org/10.1016/j.optlastec.2024.111666

2025-01-01

Abstract:Fusing infrared and visible imagery aims to harness their complementary spectral data, enhancing output image quality, sharpness, and content. However, convolutional neural networks (CNNs) are not capable of capturing long-range dependencies, while the Transformer architecture faces the challenge of consuming huge computational resources. To address these critical challenges, this paper proposes a novel framework named MBHFuse, which employs a multi-branch heterogeneous global and local image fusion approach. To effectively extract global and local features, we design a multi-branch heterogeneous encoder module and introduce a differential convolution amplification module (DCAM) to further extract complementary information. Additionally, we devise a new loss function, incorporating multi-branch feature decomposition loss, intensity loss, gradient loss, mean squared error loss, and structural similarity loss, for training the proposed MBHFuse model. Through extensive experiments on public datasets, we demonstrate that the proposed framework outperforms other state-of-the-art (SOTA) methods in both qualitative and quantitative evaluations. Our code will be available at https://github.com/sunyichen1994/MBHFuse.