Abstract:Objectives: To develop a deep learning model combining CT scans and clinical information to predict overall survival in advanced hepatocellular carcinoma (HCC). Methods: This retrospective study included immunotherapy-treated advanced HCC patients from 52 multi-national in-house centers between 2018 and 2022. A multi-modal prognostic model using baseline and the first follow-up CT images and 7 clinical variables was proposed. A convolutional-recurrent neural network (CRNN) was developed to extract spatial-temporal information from automatically selected representative 2D CT slices to provide a radiological score, then fused with a Cox-based clinical score to provide the survival risk. The model's effectiveness was assessed using a time-dependent area under the receiver operating curve (AUC), and risk group stratification using the log-rank test. Prognostic performances of multi-modal inputs were compared to models of missing modality, and the size-based RECIST criteria. Results: Two-hundred seven patients (mean age, 61 years ± 12 [SD], 180 men) were included. The multi-modal CRNN model reached the AUC of 0.777 and 0.704 of 1-year overall survival predictions in the validation and test sets. The model achieved significant risk stratification in validation (hazard ratio [HR] = 3.330, p = 0.008), and test sets (HR = 2.024, p = 0.047) based on the median risk score of the training set. Models with missing modalities (the single-modal imaging-based model and the model incorporating only baseline scans) can still achieve favorable risk stratification performance (all p < 0.05, except for one, p = 0.053). Moreover, results proved the superiority of the deep learning-based model to the RECIST criteria. Conclusion: Deep learning analysis of CT scans and clinical data can offer significant prognostic insights for patients with advanced HCC. Critical relevance statement: The established model can help monitor patients' disease statuses and identify those with poor prognosis at the time of first follow-up, helping clinicians make informed treatment decisions, as well as early and timely interventions. Key points: An AI-based prognostic model was developed for advanced HCC using multi-national patients. The model extracts spatial-temporal information from CT scans and integrates it with clinical variables to prognosticate. The model demonstrated superior prognostic ability compared to the conventional size-based RECIST method.

Multimodal Fusion with Cross-attention Transformer for HCC Early Recurrence Prediction from Multi-Phase CT and Clinical Data

A Self-Supervised Transformer-Based Model for Early Recurrence Prediction of HCC with Multimodal MRI

Deep Fusion Models of Multi-Phase CT and Selected Clinical Data for Preoperative Prediction of Early Recurrence in Hepatocellular Carcinoma.

A Transformer-Based Model for Preoperative Early Recurrence Prediction of Hepatocellular Carcinoma with Muti-modality MRI

A Unified Framework for Preoperative Early Recurrence Prediction of Hepatocellular Carcinoma with Multi-phase CT Images

Deep Learning-Based Radiomics Models for Early Recurrence Prediction of Hepatocellular Carcinoma with Multi-phase CT Images and Clinical Data.

CMIR: A Unified Cross-Modality Framework for Preoperative Accurate Prediction of Microvascular Invasion in Hepatocellular Carcinoma.

Deep Learning-based Risk Prediction Model for Recurrence-free Survival in Patients with Hepatocellular Carcinoma Using Multi-phase CT Image

Phase Attention Model for Prediction of Early Recurrence of Hepatocellular Carcinoma With Multi-Phase CT Images and Clinical Data

Prediction model of early recurrence of multimodal hepatocellular carcinoma with tensor fusion

Self-supervised Multi-Modal Feature Fusion for Predicting Early Recurrence of Hepatocellular Carcinoma

CT-Based Radiomics for the Recurrence Prediction of Hepatocellular Carcinoma after Surgical Resection

MFCSA-CAT: a Multimodal Fusion Method for Cancer Survival Analysis Based on Cross-Attention Transformer

Multi-modal Fusion Network with Intra- and Inter-Modality Attention for Prognosis Prediction in Breast Cancer

Cross-modality Attention-based Multimodal Fusion for Non-small Cell Lung Cancer (NSCLC) Patient Survival Prediction

Enhancing preoperative diagnosis of microvascular invasion in hepatocellular carcinoma: domain-adaptation fusion of multi-phase CT images

An Innovative and Efficient Diagnostic Prediction Flow for Head and Neck Cancer: A Deep Learning Approach for Multi-Modal Survival Analysis Prediction Based on Text and Multi-Center PET/CT Images

Hybrid CNN and Low-Complexity Transformer Network with Attention-Based Feature Fusion for Predicting Lung Cancer Tumor after Neoadjuvant Chemoimmunotherapy

Survival Prediction Via Hierarchical Multimodal Co-Attention Transformer: A Computational Histology-Radiology Solution.

CT-based multimodal deep learning for non-invasive overall survival prediction in advanced hepatocellular carcinoma patients treated with immunotherapy

Spatiotemporal Attention for Early Prediction of Hepatocellular Carcinoma Based on Longitudinal Ultrasound Images