Abstract:Objectives: To develop a deep learning model combining CT scans and clinical information to predict overall survival in advanced hepatocellular carcinoma (HCC). Methods: This retrospective study included immunotherapy-treated advanced HCC patients from 52 multi-national in-house centers between 2018 and 2022. A multi-modal prognostic model using baseline and the first follow-up CT images and 7 clinical variables was proposed. A convolutional-recurrent neural network (CRNN) was developed to extract spatial-temporal information from automatically selected representative 2D CT slices to provide a radiological score, then fused with a Cox-based clinical score to provide the survival risk. The model's effectiveness was assessed using a time-dependent area under the receiver operating curve (AUC), and risk group stratification using the log-rank test. Prognostic performances of multi-modal inputs were compared to models of missing modality, and the size-based RECIST criteria. Results: Two-hundred seven patients (mean age, 61 years ± 12 [SD], 180 men) were included. The multi-modal CRNN model reached the AUC of 0.777 and 0.704 of 1-year overall survival predictions in the validation and test sets. The model achieved significant risk stratification in validation (hazard ratio [HR] = 3.330, p = 0.008), and test sets (HR = 2.024, p = 0.047) based on the median risk score of the training set. Models with missing modalities (the single-modal imaging-based model and the model incorporating only baseline scans) can still achieve favorable risk stratification performance (all p < 0.05, except for one, p = 0.053). Moreover, results proved the superiority of the deep learning-based model to the RECIST criteria. Conclusion: Deep learning analysis of CT scans and clinical data can offer significant prognostic insights for patients with advanced HCC. Critical relevance statement: The established model can help monitor patients' disease statuses and identify those with poor prognosis at the time of first follow-up, helping clinicians make informed treatment decisions, as well as early and timely interventions. Key points: An AI-based prognostic model was developed for advanced HCC using multi-national patients. The model extracts spatial-temporal information from CT scans and integrates it with clinical variables to prognosticate. The model demonstrated superior prognostic ability compared to the conventional size-based RECIST method.

Multitask Learning of Longitudinal Circulating Biomarkers and Clinical Outcomes: Identification of Optimal Machine-Learning and Deep-Learning Models

Improving Prediction of Survival and Progression in Metastatic Non-Small Cell Lung Cancer Following Immunotherapy Through Machine Learning of Circulating Tumor DNA Dynamics

Improving Prediction of Survival and Progression in Metastatic Non-Small Cell Lung Cancer After Immunotherapy Through Machine Learning of Circulating Tumor DNA

A longitudinal circulating tumor DNA-based model associated with survival in metastatic non-small-cell lung cancer

Predicting survival and trial outcome in non-small cell lung cancer integrating tumor and blood markers kinetics with machine learning

Multimodal Integration of Longitudinal Noninvasive Diagnostics for Survival Prediction in Immunotherapy Using Deep Learning

Long short-term memory model – A deep learning approach for medical data with irregularity in cancer predication with tumor markers

A multi-omics-based serial deep learning approach to predict clinical outcomes of single-agent anti-PD-1/PD-L1 immunotherapy in advanced stage non-small-cell lung cancer

Radiotranscriptomics of non-small cell lung carcinoma for assessing high-level clinical outcomes using a machine learning-derived multi-modal signature

Deep-CR MTLR: a Multi-Modal Approach for Cancer Survival Prediction with Competing Risks

DeepMMSA: A Novel Multimodal Deep Learning Method for Non-small Cell Lung Cancer Survival Analysis

18F‑FDG PET/CT Based Radiomics Features Improve Prediction of Prognosis: Multiple Machine Learning Algorithms and Multimodality Applications for Multiple Myeloma

Deep learning to estimate durable clinical benefit and prognosis from patients with non-small cell lung cancer treated with PD-1/PD-L1 blockade

Evaluating Prognostic Value of Dynamics of Circulating Lactate Dehydrogenase in Colorectal Cancer Using Modeling and Machine Learning.

Machine learning applications in breast cancer survival and therapeutic outcome prediction based on multi-omic analysis

Long-term cancer survival prediction using multimodal deep learning

CT-based multimodal deep learning for non-invasive overall survival prediction in advanced hepatocellular carcinoma patients treated with immunotherapy

Predicting the efficacy of immune checkpoint inhibitors monotherapy in advanced non-small cell lung cancer: a machine learning method based on multidimensional data

Deep Learning Predicts Lung Cancer Treatment Response from Serial Medical Imaging

Multimodal Data Integration Using Deep Learning Predicts Overall Survival of Patients with Glioma

Machine learning-informed liquid-liquid phase separation for personalized breast cancer treatment assessment