FM-APP: Foundation Model for Any Phenotype Prediction Via Fmri to Smri Knowledge Transfer
Zhibin He,Wuyang Li,Yifan Liu,Xinyu Liu,Junwei Han,Tuo Zhang,Yixuan Yuan
DOI: https://doi.org/10.1109/tmi.2024.3506734
IF: 10.6
2024-01-01
IEEE Transactions on Medical Imaging
Abstract:Predicting individual-level non-neuroimaging phenotypes (e.g., fluid intelligence) using brain imaging data is a fundamental goal of neuroscience. Recent research has focused on utilizing high-cost functional magnetic resonance imaging (fMRI) to predict phenotypes seen during training. However, these methods 1) only consider predicting seen phenotypes, failing to achieve zero-shot inference for unseen phenotypes; 2) overlook the knowledge transfer from fMRI to structural MRI (sMRI), missing out on utilizing cost-effective sMRI for accurate predictions. To address these challenges, we propose a Foundational Model for Any Phenotype Prediction via fMRI to sMRI knowledge transfer (FM-APP), consisting of a Phenotypes Text Memory Bank (PTMB) module, Any Phenotype Prediction (APP) module, and fMRI to sMRI Knowledge Transfer (F2SKT) module. Our proposed FM-APP adapts to downstream tasks by generating regressor parameters instead of fine-tuning the model itself. Specifically, to retain important clues from seen phenotype descriptions, PTMB utilizes the BiomedCLIP model to store semantic features of seen phenotypes. To achieve any phenotype prediction, the APP introduces a regressor synthesizer for zero-shot inference. Additionally, to improve sMRI prediction accuracy while preserving its cost advantage, the F2SKT uses the PTMB to construct phenotype active maps, guiding adaptive knowledge transfer from fMRI to sMRI. Experiments on the Human Connectome Project (HCP) and HCP Aging datasets demonstrate our approach outperforms state-of-the-art methods, showcasing strong zero-shot inference capabilities and providing a novel framework for analyzing brain structure and phenotypes. Our code: https://github.com/ZhibinHe/FM-APP.