Abstract:Objectives: To develop an ensemble multi-task deep learning (DL) framework for automatic and simultaneous detection, segmentation, and classification of primary bone tumors (PBTs) and bone infections based on multi-parametric MRI from multi-center. Methods: This retrospective study divided 749 patients with PBTs or bone infections from two hospitals into a training set (N = 557), an internal validation set (N = 139), and an external validation set (N = 53). The ensemble framework was constructed using T1-weighted image (T1WI), T2-weighted image (T2WI), and clinical characteristics for binary (PBTs/bone infections) and three-category (benign/intermediate/malignant PBTs) classification. The detection and segmentation performances were evaluated using Intersection over Union (IoU) and Dice score. The classification performance was evaluated using the receiver operating characteristic (ROC) curve and compared with radiologist interpretations. Result: On the external validation set, the single T1WI-based and T2WI-based multi-task models obtained IoUs of 0.71 ± 0.25/0.65 ± 0.30 for detection and Dice scores of 0.75 ± 0.26/0.70 ± 0.33 for segmentation. The framework achieved AUCs of 0.959 (95%CI, 0.955-1.000)/0.900 (95%CI, 0.773-0.100) and accuracies of 90.6% (95%CI, 79.7-95.9%)/78.3% (95%CI, 58.1-90.3%) for the binary/three-category classification. Meanwhile, for the three-category classification, the performance of the framework was superior to that of three junior radiologists (accuracy: 65.2%, 69.6%, and 69.6%, respectively) and comparable to that of two senior radiologists (accuracy: 78.3% and 78.3%). Conclusion: The MRI-based ensemble multi-task framework shows promising performance in automatically and simultaneously detecting, segmenting, and classifying PBTs and bone infections, which was preferable to junior radiologists. Clinical relevance statement: Compared with junior radiologists, the ensemble multi-task deep learning framework effectively improves differential diagnosis for patients with primary bone tumors or bone infections. This finding may help physicians make treatment decisions and enable timely treatment of patients. Key points: • The ensemble framework fusing multi-parametric MRI and clinical characteristics effectively improves the classification ability of single-modality models. • The ensemble multi-task deep learning framework performed well in detecting, segmenting, and classifying primary bone tumors and bone infections. • The ensemble framework achieves an optimal classification performance superior to junior radiologists' interpretations, assisting the clinical differential diagnosis of primary bone tumors and bone infections.

Impact of metadata in multimodal classification of bone tumours

Multimodal Multitask Representation Learning for Pathology Biobank Metadata Prediction

Development and evaluation of machine learning models based on X-ray radiomics for the classification and differentiation of malignant and benign bone tumors

A deep learning model to enhance the classification of primary bone tumors based on incomplete multimodal images in X-ray, CT, and MRI

Integration of Multimodal Data for Breast Cancer Classification Using a Hybrid Deep Learning Method

Deep bone oncology Diagnostics: Computed tomography based Machine learning for detection of bone tumors from breast cancer metastasis

Multimodal Data-Driven Segmentation of Bone Metastasis Lesions in SPECT Bone Scans Using Deep Learning

Deep learning algorithm-based multimodal MRI radiomics and pathomics data improve prediction of bone metastases in primary prostate cancer

A radiograph-based deep learning model improves radiologists' performance for classification of histological types of primary bone tumors: A multicenter study

Recommender-based bone tumour classification with radiographs—a link to the past

Automated Diagnosis of Bone Metastasis by Classifying Bone Scintigrams Using a Self-defined Deep Learning Model

Multimodal Brain Tumor Classification Using Deep Learning and Robust Feature Selection: A Machine Learning Application for Radiologists

Multimodal integration of image, epigenetic and clinical data to predict BRAF mutation status in melanoma

Medical Multimodal Classifiers Under Scarce Data Condition

A deep learning–machine learning fusion approach for the classification of benign, malignant, and intermediate bone tumors

Automatic detection, segmentation, and classification of primary bone tumors and bone infections using an ensemble multi-task deep learning framework on multi-parametric MRIs: a multi-center study

Automatic classification of prostate MR series type using image content and metadata

Deep learning for tumor classification in imaging mass spectrometry

An enhanced AlexNet-Based model for femoral bone tumor classification and diagnosis using magnetic resonance imaging

Deep Representation Learning of Tissue Metabolome and Computed Tomography Images Annotates Non-invasive Classification and Prognosis Prediction of NSCLC

A multimodal machine learning model for the stratification of breast cancer risk