Abstract:Multimodal image fusion has recently garnered increasing interest in the field of remote sensing. By leveraging the complementary information in different modalities, the fused results may be more favorable in characterizing objects of interest, thereby increasing the chance of a more comprehensive and accurate perception of the scene. Unfortunately, most existing fusion methods tend to extract modality-specific features independently without considering intermodal alignment and complementarity, leading to a suboptimal fusion process. To address this issue, we propose a novel interactive generative adversarial network (IG-GAN), for the task of multimodal image fusion. IG-GAN comprises guided dual streams tailored for enhanced learning of details and content, as well as cross-modal consistency. Specifically, a details-guided interactive running-in module (GIR1) and a content-guided interactive running-in module (GIR2) are developed, with the stronger modality serving as guidance for detail richness or content integrity, and the weaker one assisting. To fully integrate multigranularity features from dual-modality, a hierarchical fusion and reconstruction branch is established. Specifically, a shallow interactive fusion (SIF) module followed by a multilevel interactive fusion (MIF) module is designed to aggregate multilevel local and long-range features. Concerning feature decoding and fused image generation, a high-level interactive fusion and reconstruction module (HRM) is further developed. In addition, to empower the fusion network to generate fused images with complete content, sharp edges, and high fidelity without supervision, a loss function facilitating the mutual game between the generator and two discriminators is also formulated. Comparative experiments with 14 state-of-the-art methods are conducted on three datasets. Qualitative and quantitative results indicate that IG-GAN exhibits obvious superiority in terms of both visual effect and quantitative metrics. Moreover, experiments on two RGB-IR object detection datasets are also conducted, which demonstrate that IG-GAN can enhance the accuracy of object detection by integrating complementary information from different modalities.

Hybrid Genetic Algorithm (GA)-based neural network for multispectral image fusion

Hyperspectral and Multispectral Image Fusion Based on Deep Attention Network.

Multi-source Image Fusion Algorithm Based on Cellular Neural Networks with Genetic Algorithm

Neural network image fusion with PCA preprocessing

GASSF-Net: Geometric Algebra Based Spectral-Spatial Hierarchical Fusion Network for Hyperspectral and LiDAR Image Classification

Design of Intelligent Spectral Selection and Multispectral Imaging Fusion Algorithm Based on Deep Neural Network Model

Hyperspectral and Multispectral Image Fusion by Deep Neural Network in a Self-Supervised Manner

Multigrained Attention Network for Infrared and Visible Image Fusion

An Unmixing-Based Multi-Attention GAN for Unsupervised Hyperspectral and Multispectral Image Fusion

A Remote Sensing Image Fusion Method Combining Low-Level Visual Features and Parameter-Adaptive Dual-Channel Pulse-Coupled Neural Network

IG-GAN: Interactive Guided Generative Adversarial Networks for Multimodal Image Fusion

Target Fusion Algorithm for Remote Sensing Image Recognition

Dual-stream GNN fusion network for hyperspectral classification

A novel image fusion method based on SGNN

A Hybrid Algorithm of the BP‐ANN/GA for Classification of Urban Terrain Surfaces with Fused Data of Landsat ETM+ and ERS‐2 SAR

Unsupervised Hyperspectral and Multispectral Images Fusion Based on Nonlinear Variational Probabilistic Generative Model

Multiple Classifier Fusion in High-resolution Remote Sensing Based on Genetic Algorithms

A Multifocus Image Fusion Algorithm for Adaptive Genetic Search

Attention Multihop Graph and Multiscale Convolutional Fusion Network for Hyperspectral Image Classification

UMAG-Net: A New Unsupervised Multiattention-Guided Network for Hyperspectral and Multispectral Image Fusion