Abstract:Objectives This study aimed to develop a deep learning radiomic model using multimodal imaging to differentiate benign and malignant breast tumours. Methods Multimodality imaging data, including ultrasonography (US), mammography (MG), and magnetic resonance imaging (MRI), from 322 patients (112 with benign breast tumours and 210 with malignant breast tumours) with histopathologically confirmed breast tumours were retrospectively collected between December 2018 and May 2023. Based on multimodal imaging, the experiment was divided into three parts: traditional radiomics, deep learning radiomics, and feature fusion. We tested the performance of seven classifiers, namely, SVM, KNN, random forest, extra trees, XGBoost, LightGBM, and LR, on different feature models. Through feature fusion using ensemble and stacking strategies, we obtained the optimal classification model for benign and malignant breast tumours. Results In terms of traditional radiomics, the ensemble fusion strategy achieved the highest accuracy, AUC, and specificity, with values of 0.892, 0.942 [0.886–0.996], and 0.956 [0.873–1.000], respectively. The early fusion strategy with US, MG, and MRI achieved the highest sensitivity of 0.952 [0.887–1.000]. In terms of deep learning radiomics, the stacking fusion strategy achieved the highest accuracy, AUC, and sensitivity, with values of 0.937, 0.947 [0.887–1.000], and 1.000 [0.999–1.000], respectively. The early fusion strategies of US+MRI and US+MG achieved the highest specificity of 0.954 [0.867–1.000]. In terms of feature fusion, the ensemble and stacking approaches of the late fusion strategy achieved the highest accuracy of 0.968. In addition, stacking achieved the highest AUC and specificity, which were 0.997 [0.990–1.000] and 1.000 [0.999–1.000], respectively. The traditional radiomic and depth features of US+MG + MR achieved the highest sensitivity of 1.000 [0.999–1.000] under the early fusion strategy. Conclusion This study demonstrated the potential of integrating deep learning and radiomic features with multimodal images. As a single modality, MRI based on radiomic features achieved greater accuracy than US or MG. The US and MG models achieved higher accuracy with transfer learning than the single-mode or radiomic models. The traditional radiomic and depth features of US+MG + MR achieved the highest sensitivity under the early fusion strategy, showed higher diagnostic performance, and provided more valuable information for differentiation between benign and malignant breast tumours.

Combining Radiomics and Autoencoders to Distinguish Benign and Malignant Breast Tumors on US Images

Deep Learning-Based Automatic Segmentation for Size and Volumetric Measurement of Breast Cancer on Magnetic Resonance Imaging.

Influence of the Computer-Aided Decision Support System Design on Ultrasound-Based Breast Cancer Classification

Multi-region radiomics for artificially intelligent diagnosis of breast cancer using multimodal ultrasound

Radiomics with Attribute Bagging for Breast Tumor Classification Using Multimodal Ultrasound Images

Dual-Intended Deep Learning Model for Breast Cancer Diagnosis in Ultrasound Imaging

Deep learning radiomics based on multimodal imaging for distinguishing benign and malignant breast tumours

Multimodality Computer-Aided Breast Cancer Diagnosis with FFDM and DCE-MRI

Automatic Breast Lesion Classification by Joint Neural Analysis of Mammography and Ultrasound

Computer-aided Diagnosis Applied to US of Solid Breast Nodules by Using Neural Networks

Machine learning-based discrimination of benign and malignant breast lesions on US: The contribution of shear-wave elastography

Predicting Breast Tumor Malignancy Using Deep ConvNeXt Radiomics and Quality-Based Score Pooling in Ultrasound Sequences

Discrimination of Breast Cancer Based on Ultrasound Images and Convolutional Neural Network

Computer-aided detection of cancer in automated 3-D breast ultrasound

Diagnosis of Breast Cancer Using Radiomics Models Built Based on Dynamic Contrast Enhanced MRI Combined With Mammography

Fusion deep learning approach combining diffuse optical tomography and ultrasound for improving breast cancer classification.

Ensemble Transfer Learning of Elastography and B-mode Breast Ultrasound Images

Real‐time breast lesion classification combining diffuse optical tomography frequency domain data and BI‐RADS assessment

Evaluating Autoencoders for Dimensionality Reduction of MRI-derived Radiomics and Classification of Malignant Brain Tumors

Automated multiparametric localization of prostate cancer based on B-mode, shear-wave elastography, and contrast-enhanced ultrasound radiomics

Predicting the malignancy of extremity soft-tissue tumors by an ultrasound-based radiomics signature