Multi-modal AI for comprehensive breast cancer prognostication
Jan Witowski,Ken Zeng,Joseph Cappadona,Jailan Elayoubi,Elena Diana Chiru,Nancy Chan,Young-Joon Kang,Frederick Howard,Irina Ostrovnaya,Carlos Fernandez-Granda,Freya Schnabel,Ugur Ozerdem,Kangning Liu,Zoe Steinsnyder,Nitya Thakore,Mohammad Sadic,Frank Yeung,Elisa Liu,Theodore Hill,Benjamin Swett,Danielle Rigau,Andrew Clayburn,Valerie Speirs,Marcus Vetter,Lina Sojak,Simone Muenst Soysal,Daniel Baumhoer,Khalil Choucair,Yu Zong,Lina Daoud,Anas Saad,Waleed Abdulsattar,Rafic Beydoun,Jia-Wern Pan,Haslina Makmur,Soo-Hwang Teo,Linda Ma Pak,Victor Angel,Dovile Zilenaite-Petrulaitiene,Arvydas Laurinavicius,Natalie Klar,Brian D. Piening,Carlo Bifulco,Sun-Young Jun,Jae Pak Yi,Su Hyun Lim,Adam Brufsky,Francisco J. Esteva,Lajos Pusztai,Yann LeCun,Krzysztof J. Geras
2024-10-29
Abstract:Treatment selection in breast cancer is guided by molecular subtypes and clinical characteristics. Recurrence risk assessment plays a crucial role in personalizing treatment. Current methods, including genomic assays, have limited accuracy and clinical utility, leading to suboptimal decisions for many patients. We developed a test for breast cancer patient stratification based on digital pathology and clinical characteristics using novel AI methods. Specifically, we utilized a vision transformer-based pan-cancer foundation model trained with self-supervised learning to extract features from digitized H&E-stained slides. These features were integrated with clinical data to form a multi-modal AI test predicting cancer recurrence and death. The test was developed and evaluated using data from a total of 8,161 breast cancer patients across 15 cohorts originating from seven countries. Of these, 3,502 patients from five cohorts were used exclusively for evaluation, while the remaining patients were used for training. Our test accurately predicted our primary endpoint, disease-free interval, in the five external cohorts (C-index: 0.71 [0.68-0.75], HR: 3.63 [3.02-4.37, p<0.01]). In a direct comparison (N=858), the AI test was more accurate than Oncotype DX, the standard-of-care 21-gene assay, with a C-index of 0.67 [0.61-0.74] versus 0.61 [0.49-0.73], respectively. Additionally, the AI test added independent information to Oncotype DX in a multivariate analysis (HR: 3.11 [1.91-5.09, p<0.01)]). The test demonstrated robust accuracy across all major breast cancer subtypes, including TNBC (C-index: 0.71 [0.62-0.81], HR: 3.81 [2.35-6.17, p=0.02]), where no diagnostic tools are currently recommended by clinical guidelines. These results suggest that our AI test can improve accuracy, extend applicability to a wider range of patients, and enhance access to treatment selection tools.
Artificial Intelligence,Computer Vision and Pattern Recognition,Image and Video Processing