Dynamic Brightness Adaptation for Robust Multi-modal Image Fusion

Yiming Sun,Bing Cao,Pengfei Zhu,Qinghua Hu

DOI: https://doi.org/10.24963/ijcai.2024/146

2024-11-07

Abstract:Infrared and visible image fusion aim to integrate modality strengths for visually enhanced, informative images. Visible imaging in real-world scenarios is susceptible to dynamic environmental brightness fluctuations, leading to texture degradation. Existing fusion methods lack robustness against such brightness perturbations, significantly compromising the visual fidelity of the fused imagery. To address this challenge, we propose the Brightness Adaptive multimodal dynamic fusion framework (BA-Fusion), which achieves robust image fusion despite dynamic brightness fluctuations. Specifically, we introduce a Brightness Adaptive Gate (BAG) module, which is designed to dynamically select features from brightness-related channels for normalization, while preserving brightness-independent structural information within the source images. Furthermore, we propose a brightness consistency loss function to optimize the BAG module. The entire framework is tuned via alternating training strategies. Extensive experiments validate that our method surpasses state-of-the-art methods in preserving multi-modal image information and visual fidelity, while exhibiting remarkable robustness across varying brightness levels. Our code is available: <a class="link-external link-https" href="https://github.com/SunYM2020/BA-Fusion" rel="external noopener nofollow">this https URL</a>.

Computer Vision and Pattern Recognition

What problem does this paper attempt to address?

### What problem does this paper attempt to solve?

This paper aims to solve the robustness problem of infrared and visible - light image fusion under the change of dynamic environmental brightness. Specifically, existing methods often fail to maintain the visual fidelity and information integrity of the fused image when facing environmental brightness fluctuations. This causes the quality of the fusion result to fluctuate with the change of environmental brightness, thereby reducing the visual effect of the fused image.

#### Problem Background

1. **The Influence of Environmental Brightness Fluctuations**: Visible - light imaging in real - world scenes is easily affected by the change of dynamic environmental brightness, resulting in texture degradation.
2. **Limitations of Existing Methods**: Existing image fusion methods lack robustness to brightness changes. Especially when dealing with over - exposed or low - light images, they cannot effectively maintain the information and visual fidelity of multi - modal images.

#### Solution

To solve the above problems, the author proposes a brightness - adaptive multi - modal dynamic fusion framework named **BA - Fusion**. The core components of this framework include:

1. **Brightness Adaptive Gate (BAG) Module**:
- **Brightness Normalization**: Normalize the channels related to brightness to eliminate the influence of brightness.
- **Dynamically Select Channels**: Dynamically select the feature channels most relevant to brightness changes in a data - driven manner while retaining the structure information unrelated to brightness.

2. **Brightness Consistency Loss Function**:
- Ensure that under different brightness perturbations, the frequency - domain brightness representation of the fusion result is consistent with that of the normal fusion result, thereby constraining the learning process of the BAG module.

3. **Alternating Training Strategy**:
- Optimize the BAG module through the alternating training strategy, gradually establish the connection between brightness changes and feature channels, and ensure that the model has the brightness - adaptive robust fusion ability.

#### Main Contributions

- Propose a brightness - adaptive dynamic image fusion framework, which can effectively alleviate the problem of unstable fusion effects caused by environmental brightness fluctuations.
- Introduce the brightness - adaptive gate module and establish the correspondence between the input image brightness and the channel feature representation.
- Dynamically balance the advantages of visible - light and infrared modalities in terms of texture details and contrast. Experiments have proved its superior performance on multiple infrared - visible - light data sets.

### Summary

This paper solves the problem of insufficient robustness of existing image fusion methods in the environment of dynamic brightness changes by proposing the BA - Fusion framework, and realizes more stable and high - quality multi - modal image fusion.