Cross-modal autoencoder framework learns holistic representations of cardiovascular state

Adityanarayanan Radhakrishnan,Sam F Friedman,Shaan Khurshid,Kenney Ng,Puneet Batra,Steven A Lubitz,Anthony A Philippakis,Caroline Uhler
DOI: https://doi.org/10.1038/s41467-023-38125-0
2023-04-28
Abstract:A fundamental challenge in diagnostics is integrating multiple modalities to develop a joint characterization of physiological state. Using the heart as a model system, we develop a cross-modal autoencoder framework for integrating distinct data modalities and constructing a holistic representation of cardiovascular state. In particular, we use our framework to construct such cross-modal representations from cardiac magnetic resonance images (MRIs), containing structural information, and electrocardiograms (ECGs), containing myoelectric information. We leverage the learned cross-modal representation to (1) improve phenotype prediction from a single, accessible phenotype such as ECGs; (2) enable imputation of hard-to-acquire cardiac MRIs from easy-to-acquire ECGs; and (3) develop a framework for performing genome-wide association studies in an unsupervised manner. Our results systematically integrate distinct diagnostic modalities into a common representation that better characterizes physiologic state.
What problem does this paper attempt to address?