Abstract:Multimodal image fusion has recently garnered increasing interest in the field of remote sensing. By leveraging the complementary information in different modalities, the fused results may be more favorable in characterizing objects of interest, thereby increasing the chance of a more comprehensive and accurate perception of the scene. Unfortunately, most existing fusion methods tend to extract modality-specific features independently without considering intermodal alignment and complementarity, leading to a suboptimal fusion process. To address this issue, we propose a novel interactive generative adversarial network (IG-GAN), for the task of multimodal image fusion. IG-GAN comprises guided dual streams tailored for enhanced learning of details and content, as well as cross-modal consistency. Specifically, a details-guided interactive running-in module (GIR1) and a content-guided interactive running-in module (GIR2) are developed, with the stronger modality serving as guidance for detail richness or content integrity, and the weaker one assisting. To fully integrate multigranularity features from dual-modality, a hierarchical fusion and reconstruction branch is established. Specifically, a shallow interactive fusion (SIF) module followed by a multilevel interactive fusion (MIF) module is designed to aggregate multilevel local and long-range features. Concerning feature decoding and fused image generation, a high-level interactive fusion and reconstruction module (HRM) is further developed. In addition, to empower the fusion network to generate fused images with complete content, sharp edges, and high fidelity without supervision, a loss function facilitating the mutual game between the generator and two discriminators is also formulated. Comparative experiments with 14 state-of-the-art methods are conducted on three datasets. Qualitative and quantitative results indicate that IG-GAN exhibits obvious superiority in terms of both visual effect and quantitative metrics. Moreover, experiments on two RGB-IR object detection datasets are also conducted, which demonstrate that IG-GAN can enhance the accuracy of object detection by integrating complementary information from different modalities.

MHW-GAN: Multidiscriminator Hierarchical Wavelet Generative Adversarial Network for Multimodal Image Fusion.

IG-GAN: Interactive Guided Generative Adversarial Networks for Multimodal Image Fusion

Transformer Based Conditional GAN for Multimodal Image Fusion

An attention-guided and wavelet-constrained generative adversarial network for infrared and visible image fusion

MFF-GAN: An unsupervised generative adversarial network with adaptive and gradient joint constraints for multi-focus image fusion

AT-GAN: A generative adversarial network with attention and transition for infrared and visible image fusion

Image Fusion Via Hierarchical Extractor and Maximum Feature Distribution Learning

JADD-GAN: A Joint Attention Generative Adversarial Data Fusion Network for Object Detection and Tracking.

GANMcC: A Generative Adversarial Network With Multiclassification Constraints for Infrared and Visible Image Fusion

An Adaptive Multi-Perceptual Implicit Sampling for Hyperspectral and Multispectral Remote Sensing Image Fusion

DDF-GAN: A Generative Adversarial Network with Dual-Discriminator for Multi-Focus Image Fusion

MDRDA-GAN: A Multi-discriminator Generative Adversarial Network Based on Residual Dense Attention for Remote Sensing Image Fusion

MEF-GAN: Multi-Exposure Image Fusion via Generative Adversarial Networks

Research on Multimodal Image Fusion Target Detection Algorithm Based on Generative Adversarial Network

Fusion-UDCGAN: Multifocus Image Fusion via a U-Type Densely Connected Generation Adversarial Network

Shortwave Infrared and Visible Light Image Fusion Method Based on Dual Discriminator GAN

Multi-focus images fusion via residual generative adversarial network

Multi-Level Interactive Fusion Network Based on Adversarial Learning for Fusion Classification of Hyperspectral and Lidar Data

Mutual-Guided Dynamic Network for Image Fusion

GAN-HA: A generative adversarial network with a novel heterogeneous dual-discriminator network and a new attention-based fusion strategy for infrared and visible image fusion