Abstract:Multi-modal image fusion synthesizes information from multiple sources into a single image, facilitating downstream tasks such as semantic segmentation. Current approaches primarily focus on acquiring informative fusion images at the visual display stratum through intricate mappings. Although some approaches attempt to jointly optimize image fusion and downstream tasks, these efforts often lack direct guidance or interaction, serving only to assist with a predefined fusion loss. To address this, we propose an “Unfolding Attribution Analysis Fusion network” (UAAFusion), using attribution analysis to tailor fused images more effectively for semantic segmentation, enhancing the interaction between the fusion and segmentation. Specifically, we utilize attribution analysis techniques to explore the contributions of semantic regions in the source images to task discrimination. At the same time, our fusion algorithm incorporates more beneficial features from the source images, thereby allowing the segmentation to guide the fusion process. Our method constructs a model-driven unfolding network that uses optimization objectives derived from attribution analysis, with an attribution fusion loss calculated from the current state of the segmentation network. We also develop a new pathway function for attribution analysis, specifically tailored to the fusion tasks in our unfolding network. An attribution attention mechanism is integrated at each network stage, allowing the fusion network to prioritize areas and pixels crucial for high-level recognition tasks. Additionally, to mitigate the information loss in traditional unfolding networks, a memory augmentation module is incorporated into our network to improve the information flow across various network layers. Extensive experiments demonstrate our method’s superiority in image fusion and applicability to semantic segmentation. The code is available at https://github.com/HaowenBai/UAAFusion.

Fusing heterogeneous information for multi-modal attributed network embedding

Flexible Attributed Network Embedding.

Fast Attributed Multiplex Heterogeneous Network Embedding.

Attributed Multi-layer Network Embedding.

Unsupervised Attributed Network Embedding Via Cross Fusion

Representation Learning for Attributed Multiplex Heterogeneous Network

Attention Based Meta Path Fusion for Heterogeneous Information Network Embedding

Hierarchical Label with Imbalance and Attributed Network Structure Fusion for Network Embedding

Multimodal Deep Network Embedding with Integrated Structure and Attribute Information.

A 2 CMHNE: Attention-Aware Collaborative Multimodal Heterogeneous Network Embedding

Heterogeneous Attributed Network Embedding with Graph Convolutional Networks

AHNG: Representation Learning on Attributed Heterogeneous Network

Deep Unfolding Multi-modal Image Fusion Network Via Attribution Analysis

Heterogeneous-attributes Enhancement Deep Framework for Network Embedding

Attributed Heterogeneous Network Fusion via Collaborative Matrix Tri-factorization

Dynamic Heterogeneous Attributed Network Embedding

An interlayer feature fusion-based heterogeneous graph neural network

Fusing Attributed and Topological Global-Relations for Network Embedding.

A 2 CMHNE

Network embedding by fusing multimodal contents and links

Embedding Representation of Academic Heterogeneous Information Networks Based on Federated Learning