Abstract:Abstract In radiation oncology, predicting patient risk stratification allows specialization of therapy intensification as well as selecting between systemic and regional treatments, all of which helps to improve patient outcome and quality of life. Deep learning offers an advantage over traditional radiomics for medical image processing by learning salient features from training data originating from multiple datasets. However, while their large capacity allows to combine high-level medical imaging data for outcome prediction, they lack generalization to be used across institutions. In this work, a pseudo-volumetric convolutional neural network with a deep preprocessor module and self-attention (PreSANet) is proposed for the prediction of distant metastasis, locoregional recurrence, and overall survival occurrence probabilities within the 10 year follow-up time frame for head and neck cancer patients with squamous cell carcinoma. The model is capable of processing multi-modal inputs of variable scan length, as well as integrating patient data in the prediction model. These proposed architectural features and additional modalities all serve to extract additional information from the available data when availability to additional samples is limited. This model was trained on the public Cancer Imaging Archive Head–Neck-PET–CT dataset consisting of 298 patients undergoing curative radio/chemo-radiotherapy and acquired from 4 different institutions. The model was further validated on an internal retrospective dataset with 371 patients acquired from one of the institutions in the training dataset. An extensive set of ablation experiments were performed to test the utility of the proposed model characteristics, achieving an AUROC of $$80\%$$ 80 % , $$80\%$$ 80 % and $$82\%$$ 82 % for DM, LR and OS respectively on the public TCIA Head–Neck-PET–CT dataset. External validation was performed on a retrospective dataset with 371 patients, achieving $$69\%$$ 69 % AUROC in all outcomes. To test for model generalization across sites, a validation scheme consisting of single site-holdout and cross-validation combining both datasets was used. The mean accuracy across 4 institutions obtained was $$72\%$$ 72 % , $$70\%$$ 70 % and $$71\%$$ 71 % for DM, LR and OS respectively. The proposed model demonstrates an effective method for tumor outcome prediction for multi-site, multi-modal combining both volumetric data and structured patient clinical data.

Multi-task Deep Learning for Joint Tumor Segmentation and Outcome Prediction in Head and Neck Cancer.

Radiomics-Enhanced Deep Multi-task Learning for Outcome Prediction in Head and Neck Cancer

Multimodal PET/CT Tumour Segmentation and Prediction of Progression-Free Survival using a Full-Scale UNet with Attention

Segmentation-Free Outcome Prediction in Head and Neck Cancer: Deep Learning-based Feature Extraction from Multi-Angle Maximum Intensity Projections (MA-MIPs) of PET Images

Deep Learning Based Time-to-event Analysis with PET, CT and Joint PET/CT for Head and Neck Cancer Prognosis

Multi-modal co-learning with attention mechanism for head and neck tumor segmentation on 18FDG PET-CT

Multitask Learning with Convolutional Neural Networks and Vision Transformers Can Improve Outcome Prediction for Head and Neck Cancer Patients

Joint nnU-Net and Radiomics Approaches for Segmentation and Prognosis of Head and Neck Cancers with PET/CT images

DeepMTS: Deep Multi-task Learning for Survival Prediction in Patients with Advanced Nasopharyngeal Carcinoma using Pretreatment PET/CT

Segmentation-Free Outcome Prediction from Head and Neck Cancer PET/CT Images: Deep Learning-Based Feature Extraction from Multi-Angle Maximum Intensity Projections (MA-MIPs)

Merging-Diverging Hybrid Transformer Networks for Survival Prediction in Head and Neck Cancer

Head and neck tumor segmentation convolutional neural network robust to missing PET/CT modalities using channel dropout

An Innovative and Efficient Diagnostic Prediction Flow for Head and Neck Cancer: A Deep Learning Approach for Multi-Modal Survival Analysis Prediction Based on Text and Multi-Center PET/CT Images

Multi-task multi-scale learning for outcome prediction in 3D PET images

Combining CNN and Hybrid Active Contours for Head and Neck Tumor Segmentation in CT and PET images

Comparing different CT, PET and MRI multi-modality image combinations for deep learning-based head and neck tumor segmentation

Cross-institutional outcome prediction for head and neck cancer patients using self-attention neural networks

Predicting Treatment Response from Longitudinal Images Using Multi-Task Deep Learning

Joint Lymphoma Lesion Segmentation and Prognosis Prediction from Baseline FDG-PET Images Via Multitask Convolutional Neural Networks

An Ensemble Approach for Patient Prognosis of Head and Neck Tumor Using Multimodal Data

Towards Tumour Graph Learning for Survival Prediction in Head & Neck Cancer Patients