Dynamic Graph Neural Representation Based Multi-modal Fusion Model for Cognitive Outcome Prediction in Stroke Cases

Shuting Liu,Baochang Zhang,Rong Fang,Daniel Rueckert,Veronika A. Zimmer
DOI: https://doi.org/10.1007/978-3-031-43993-3_33
2023-01-01
Abstract:The number of stroke patients is growing worldwide and half of them will suffer from cognitive impairment. Therefore, the prediction of Post-Stroke Cognitive Impairment (PSCI) becomes more and more important. However, the determinants and mechanisms of PSCI are still insufficiently understood, making this task challenging. In this paper, we propose a multi-modal graph fusion model to solve this task. First, dynamic graph neural representation is proposed to integrate multi-modal information, such as clinical data and image data, which separates them into node-level and global-level properties rather than processing them uniformly. Second, considering the variability of brain anatomy, a subject-specific undirected graph is constructed based on the connections among 131 brain anatomical regions segmented from image data, while first-order statistical features are extracted from each brain region and internal stroke lesions as node features. Meanwhile, a novel missing information compensation module is proposed to reduce the impact of missing or incomplete clinical data. In the dynamic graph neural representation, two kinds of attention mechanisms are used to encourage the model to automatically localize brain anatomical regions that are highly relevant to PSCI prediction. One is node attention established between global tabular neural representation and nodes, the other is multi-head graph self-attention which changes the static undirected graph into multiple dynamic directed graphs and optimizes the broadcasting process of the graph. The proposed method studies 418 stroke patients and achieves the best overall performance with a balanced accuracy score of 79.6% on PSCI prediction, outperforming the competing models. The code is publicly available at github.com/fightingkitty/MHGSA.
What problem does this paper attempt to address?