Abstract:Multimodal magnetic resonance imaging (MRI) provides complementary information about targets, and the segmentation of multimodal MRI is widely used as an essential preprocessing step for initial diagnosis, stage differentiation, and post-treatment efficacy evaluation in clinical situations. For the main modality or each of the modalities, it is important to enhance the visual information by modeling the connection and effectively fusing the features among them. However, the existing methods for multimodal segmentation have a drawback; they coincidentally drop information of individual modality during the fusion process. Recently, graph learning-based methods have been applied in segmentation, and these methods have achieved considerable improvements by modeling the relationships across feature regions and reasoning using global information. In this paper, we propose a graph learning-based approach to efficiently extract modality-specific features and establish regional correspondence effectively among all modalities. In detail, after projecting features into a graph domain and employing graph convolution to propagate information across all regions for learning global modality-specific features, we propose a mutual information-based graph co-attention module to learn the weight coefficients of one bipartite graph constructed by the fully connected graphs having different modalities in the graph domain and by selectively fusing the node features. Based on the deformation diagram between the spatial-graph space and our proposed graph co-attention module, we present a multimodal prior-guided segmentation framework, which uses two strategies for two clinical situations: Modality-Specific Learning Strategy and Co-Modality Learning Strategy. Besides, the improved Co-Modality Learning Strategy is used with trainable weights in the multi-task loss for the optimization of the proposed framework. We validated our proposed modules and frameworks on two multimodal MRI datasets: our private liver lesion dataset and a public prostate zone dataset. Our experimental results on both datasets prove the superiority of our proposed approaches.

Material structure segmentation method based on graph attention

Mutual Information-Based Graph Co-Attention Networks for Multimodal Prior-Guided Magnetic Resonance Imaging Segmentation

Deep Dual-Stream Network with Scale Context Selection Attention Module for Semantic Segmentation

Edge-Enhanced GCIFFNet: A Multiclass Semantic Segmentation Network Based on Edge Enhancement and Multiscale Attention Mechanism

MatSAM: Efficient Extraction of Microstructures of Materials via Visual Large Model

Material Segmentation of Multi-View Satellite Imagery

Center-environment feature models for materials image segmentation based on machine learning

Graph attention neural networks for mapping materials and molecules beyond short-range interatomic correlations

A node graph using SEM images for material properties prediction with a case study of composite material

A Deep Learning Labeling Method for Material Microstructure Image Segmentation

XANet: An Efficient Remote Sensing Image Segmentation Model Using Element-Wise Attention Enhancement and Multi-Scale Attention Fusion

Mining Sufficient Knowledge via Progressive Feature Fusion for Efficient Material Recognition

Multi-scale attention-based instance segmentation for measuring crystals with large size variation

Research on Image Semantic Segmentation Based on Hybrid Cascade Feature Fusion and Detailed Attention Mechanism

Based on cross-scale fusion attention mechanism network for semantic segmentation for street scenes

A pseudo-labeling based weakly supervised segmentation method for few-shot texture images

CEGANN: Crystal Edge Graph Attention Neural Network for multiscale classification of materials environment

Microstructural segmentation using a union of attention guided U-Net models with different color transformed images

Central Attention with Multi-Graphs for Image Annotation

Lightweight Attention Network for Very High-Resolution Image Semantic Segmentation

Stair Fusion Network With Context-Refined Attention for Remote Sensing Image Semantic Segmentation