Multimodal Deep Learning for Personalized Renal Cell Carcinoma Prognosis: Integrating CT Imaging and Clinical Data

Maryamalsadat Mahootiha,Hemin Ali Qadir,Jacob Bergsland,Ilangko Balasingham

2023-07-07

Abstract:Renal cell carcinoma represents a significant global health challenge with a low survival rate. This research aimed to devise a comprehensive deep-learning model capable of predicting survival probabilities in patients with renal cell carcinoma by integrating CT imaging and clinical data and addressing the limitations observed in prior studies. The aim is to facilitate the identification of patients requiring urgent treatment. The proposed framework comprises three modules: a 3D image feature extractor, clinical variable selection, and survival prediction. The feature extractor module, based on the 3D CNN architecture, predicts the ISUP grade of renal cell carcinoma tumors linked to mortality rates from CT images. A selection of clinical variables is systematically chosen using the Spearman score and random forest importance score as criteria. A deep learning-based network, trained with discrete LogisticHazard-based loss, performs the survival prediction. Nine distinct experiments are performed, with varying numbers of clinical variables determined by different thresholds of the Spearman and importance scores. Our findings demonstrate that the proposed strategy surpasses the current literature on renal cancer prognosis based on CT scans and clinical factors. The best-performing experiment yielded a concordance index of 0.84 and an area under the curve value of 0.8 on the test cohort, which suggests strong predictive power. The multimodal deep-learning approach developed in this study shows promising results in estimating survival probabilities for renal cell carcinoma patients using CT imaging and clinical data. This may have potential implications in identifying patients who require urgent treatment, potentially improving patient outcomes. The code created for this project is available for the public on: \href{<a class="link-external link-https" href="https://github.com/Balasingham-AI-Group/Survival_CTplusClinical" rel="external noopener nofollow">this https URL</a>}{GitHub}

Computer Vision and Pattern Recognition,Artificial Intelligence

What problem does this paper attempt to address?

### Problems the paper attempts to solve

This paper aims to solve the problem of survival prediction for patients with renal cell carcinoma (RCC). Renal cell carcinoma is a common malignant tumor in adults, accounting for about 90% of all kidney tumors, and has a relatively low survival rate. Early detection and treatment are crucial for improving patient prognosis and increasing survival rate. However, existing clinical methods have limitations in predicting the survival probability of patients with renal cell carcinoma. For example, doctors may make inaccurate predictions due to lack of experience or misinterpretation of images.

To overcome these limitations, this study proposes a multimodal deep - learning model to predict the survival probability of patients with renal cell carcinoma by integrating CT image data and clinical data. Specifically, the model includes the following three modules:

1. **3D Image Feature Extraction Module**: Based on the 3D convolutional neural network (3D CNN) architecture, extract features related to the ISUP grading from CT images. The ISUP grading system is used to assess the severity of tumors and is closely related to mortality.
2. **Clinical Variable Selection Module**: Use the Spearman correlation coefficient and random forest importance score to systematically select clinical variables.
3. **Survival Prediction Module**: A deep - learning network trained based on the discrete Logistic Hazard loss function for survival prediction.

Through the comprehensive application of these modules, this study aims to improve the accuracy of survival prediction for patients with renal cell carcinoma, thereby helping to identify patients in need of urgent treatment and improving patients' treatment effects and quality of life.