Abstract:Multimodal image fusion has recently garnered increasing interest in the field of remote sensing. By leveraging the complementary information in different modalities, the fused results may be more favorable in characterizing objects of interest, thereby increasing the chance of a more comprehensive and accurate perception of the scene. Unfortunately, most existing fusion methods tend to extract modality-specific features independently without considering intermodal alignment and complementarity, leading to a suboptimal fusion process. To address this issue, we propose a novel interactive generative adversarial network (IG-GAN), for the task of multimodal image fusion. IG-GAN comprises guided dual streams tailored for enhanced learning of details and content, as well as cross-modal consistency. Specifically, a details-guided interactive running-in module (GIR1) and a content-guided interactive running-in module (GIR2) are developed, with the stronger modality serving as guidance for detail richness or content integrity, and the weaker one assisting. To fully integrate multigranularity features from dual-modality, a hierarchical fusion and reconstruction branch is established. Specifically, a shallow interactive fusion (SIF) module followed by a multilevel interactive fusion (MIF) module is designed to aggregate multilevel local and long-range features. Concerning feature decoding and fused image generation, a high-level interactive fusion and reconstruction module (HRM) is further developed. In addition, to empower the fusion network to generate fused images with complete content, sharp edges, and high fidelity without supervision, a loss function facilitating the mutual game between the generator and two discriminators is also formulated. Comparative experiments with 14 state-of-the-art methods are conducted on three datasets. Qualitative and quantitative results indicate that IG-GAN exhibits obvious superiority in terms of both visual effect and quantitative metrics. Moreover, experiments on two RGB-IR object detection datasets are also conducted, which demonstrate that IG-GAN can enhance the accuracy of object detection by integrating complementary information from different modalities.

ICWGAN-GP: an image fusion method based on infrared compensator and wasserstein generative adversarial network with gradient penalty

An attention-guided and wavelet-constrained generative adversarial network for infrared and visible image fusion

GAN-HA: A generative adversarial network with a novel heterogeneous dual-discriminator network and a new attention-based fusion strategy for infrared and visible image fusion

GAN-GA: infrared and visible image fusion generative adversarial network based on global awareness

FusionGAN: A Generative Adversarial Network for Infrared and Visible Image Fusion.

Coupled GAN With Relativistic Discriminators for Infrared and Visible Images Fusion

Infrared/Visible Light Fire Image Fusion Method Based on Generative Adversarial Network of Wavelet-Guided Pooling Vision Transformer

AT-GAN: A generative adversarial network with attention and transition for infrared and visible image fusion

A Generative Adversarial Network for infrared and visible image fusion using adaptive dense generator and Markovian discriminator

Infrared and Visible Image Fusion via Interactive Compensatory Attention Adversarial Learning

DDGANSE: Dual-Discriminator GAN with a Squeeze-and-Excitation Module for Infrared and Visible Image Fusion

CSPA-GAN: A Cross-Scale Pyramid Attention GAN for Infrared and Visible Image Fusion

GAN-FM: Infrared and Visible Image Fusion Using GAN with Full-Scale Skip Connection and Dual Markovian Discriminators

Research on Fusion Algorithm of Infrared and Visible Image Based on Generative Adversarial Network and Attention Mechanism

Dense-FG: A Fusion GAN Model by Using Densely Connected Blocks to Fuse Infrared and Visible Images

Infrared-Visible Image Fusion Based on Semantic Guidance and Visual Perception

Infrared and Visible Image Fusion via Texture Conditional Generative Adversarial Network

A Generative Adversarial Network for Infrared and Visible Image Fusion Based on Semantic Segmentation

Fusion of Infrared and Visible Images Based on Improved Generative Adversarial Networks

Shortwave Infrared and Visible Light Image Fusion Method Based on Dual Discriminator GAN

IG-GAN: Interactive Guided Generative Adversarial Networks for Multimodal Image Fusion