Abstract:Multimodal magnetic resonance imaging (MRI) provides complementary information about targets, and the segmentation of multimodal MRI is widely used as an essential preprocessing step for initial diagnosis, stage differentiation, and post-treatment efficacy evaluation in clinical situations. For the main modality or each of the modalities, it is important to enhance the visual information by modeling the connection and effectively fusing the features among them. However, the existing methods for multimodal segmentation have a drawback; they coincidentally drop information of individual modality during the fusion process. Recently, graph learning-based methods have been applied in segmentation, and these methods have achieved considerable improvements by modeling the relationships across feature regions and reasoning using global information. In this paper, we propose a graph learning-based approach to efficiently extract modality-specific features and establish regional correspondence effectively among all modalities. In detail, after projecting features into a graph domain and employing graph convolution to propagate information across all regions for learning global modality-specific features, we propose a mutual information-based graph co-attention module to learn the weight coefficients of one bipartite graph constructed by the fully connected graphs having different modalities in the graph domain and by selectively fusing the node features. Based on the deformation diagram between the spatial-graph space and our proposed graph co-attention module, we present a multimodal prior-guided segmentation framework, which uses two strategies for two clinical situations: Modality-Specific Learning Strategy and Co-Modality Learning Strategy. Besides, the improved Co-Modality Learning Strategy is used with trainable weights in the multi-task loss for the optimization of the proposed framework. We validated our proposed modules and frameworks on two multimodal MRI datasets: our private liver lesion dataset and a public prostate zone dataset. Our experimental results on both datasets prove the superiority of our proposed approaches.

Multi-class Cosegmentation with Pairwise Active Learning.

Mutual Information-Based Graph Co-Attention Networks for Multimodal Prior-Guided Magnetic Resonance Imaging Segmentation

Unsupervised Multi-Class Co-Segmentation via Joint-Cut Over $L_{1}$ -Manifold Hyper-Graph of Discriminative Image Regions

Multi-class Object Segmentation Based on Jointly Integrating Segment-Level and Image-Level Object Priors

Saliency Based Multiple Object Cosegmentation by Ensemble MIML Learning

Weak Supervision Learning for Object Co-Segmentation

Coarse-to-fine Image Co-segmentation with Intra and Inter Rank Constraints

Image Cosegmentation via Saliency-Guided Constrained Clustering with Cosine Similarity

Cosegmentation and Cosketch by Unsupervised Learning.

Social Media Analytics and Learning

Multiple Semantic Matching on Augmented $N$ -Partite Graph for Object Co-Segmentation

Object Co-Segmentation Based on Directed Graph Clustering

Correlative multi-label multi-instance image annotation

Multi-Class Constrained Normalized Cut with Hard, Soft, Unary and Pairwise Priors and Its Applications to Object Segmentation

Deep Class-Specific Affinity-Guided Convolutional Network for Multimodal Unpaired Image Segmentation

Deep Object Co-segmentation via Spatial-Semantic Network Modulation

Unpaired Multi-modal Segmentation via Knowledge Distillation

3D Shape Co-segmentation by Combining Sparse Representation with Extreme Learning Machine.

Active Learning with Label Correlation Exploration for Multi-Label Image Classification

MULTI-LABEL IMAGE RECOGNITION WITH JOINT CLASS-AWARE MAP DISENTANGLING AND LABEL CORRELATION EMBEDDING

Learning Co-segmentation by Segment Swapping for Retrieval and Discovery