Abstract:Multimodal magnetic resonance imaging (MRI) provides complementary information about targets, and the segmentation of multimodal MRI is widely used as an essential preprocessing step for initial diagnosis, stage differentiation, and post-treatment efficacy evaluation in clinical situations. For the main modality or each of the modalities, it is important to enhance the visual information by modeling the connection and effectively fusing the features among them. However, the existing methods for multimodal segmentation have a drawback; they coincidentally drop information of individual modality during the fusion process. Recently, graph learning-based methods have been applied in segmentation, and these methods have achieved considerable improvements by modeling the relationships across feature regions and reasoning using global information. In this paper, we propose a graph learning-based approach to efficiently extract modality-specific features and establish regional correspondence effectively among all modalities. In detail, after projecting features into a graph domain and employing graph convolution to propagate information across all regions for learning global modality-specific features, we propose a mutual information-based graph co-attention module to learn the weight coefficients of one bipartite graph constructed by the fully connected graphs having different modalities in the graph domain and by selectively fusing the node features. Based on the deformation diagram between the spatial-graph space and our proposed graph co-attention module, we present a multimodal prior-guided segmentation framework, which uses two strategies for two clinical situations: Modality-Specific Learning Strategy and Co-Modality Learning Strategy. Besides, the improved Co-Modality Learning Strategy is used with trainable weights in the multi-task loss for the optimization of the proposed framework. We validated our proposed modules and frameworks on two multimodal MRI datasets: our private liver lesion dataset and a public prostate zone dataset. Our experimental results on both datasets prove the superiority of our proposed approaches.

Multi-Aspect Understanding with Cooperative Graph Attention Networks for Medical Dialogue Information Extraction

A Speaker-Aware Co-Attention Framework for Medical Dialogue Information Extraction

Mutual Information-Based Graph Co-Attention Networks for Multimodal Prior-Guided Magnetic Resonance Imaging Segmentation

Dual Edge-embedding Graph Convolutional Network for Unified Aspect-based Sentiment Analysis

BioIE: Biomedical Information Extraction with Multi-head Attention Enhanced Graph Convolutional Network

Dialogue Relation Extraction with Document-Level Heterogeneous Graph Attention Networks

Biomedical cross-sentence relation extraction via multihead attention and graph convolutional networks

DDI-MuG: Multi-aspect graphs for drug-drug interaction extraction

Dual Attention Graph Convolutional Network for Relation Extraction

Document-Level Biomedical Relation Extraction Using Graph Convolutional Network and Multihead Attention: Algorithm Development and Validation

Contextual-Aware Information Extractor with Adaptive Objective for Chinese Medical Dialogues

Position-Aware Attention Mechanism–Based Bi-graph for Dialogue Relation Extraction

Position Enhanced Mention Graph Attention Network for Dialogue Relation Extraction

Incorporating representation learning and multihead attention to improve biomedical cross-sentence n-ary relation extraction

Intelligent extraction of medical entity relationship based on graph neural network and optimization strategy

Improving Medical Dialogue Generation with Abstract Meaning Representations

A Knowledge-enhanced Two-stage Generative Framework for Medical Dialogue Information Extraction

Graph Ensemble Learning over Multiple Dependency Trees for Aspect-level Sentiment Classification.

Graph-Evolving Meta-Learning for Low-Resource Medical Dialogue Generation

Enhancing Representation in Medical Vision-Language Foundation Models via Multi-Scale Information Extraction Techniques

Medical Dialogue Generation via Dual Flow Modeling