Abstract:Objectives This study aimed to develop a deep learning radiomic model using multimodal imaging to differentiate benign and malignant breast tumours. Methods Multimodality imaging data, including ultrasonography (US), mammography (MG), and magnetic resonance imaging (MRI), from 322 patients (112 with benign breast tumours and 210 with malignant breast tumours) with histopathologically confirmed breast tumours were retrospectively collected between December 2018 and May 2023. Based on multimodal imaging, the experiment was divided into three parts: traditional radiomics, deep learning radiomics, and feature fusion. We tested the performance of seven classifiers, namely, SVM, KNN, random forest, extra trees, XGBoost, LightGBM, and LR, on different feature models. Through feature fusion using ensemble and stacking strategies, we obtained the optimal classification model for benign and malignant breast tumours. Results In terms of traditional radiomics, the ensemble fusion strategy achieved the highest accuracy, AUC, and specificity, with values of 0.892, 0.942 [0.886–0.996], and 0.956 [0.873–1.000], respectively. The early fusion strategy with US, MG, and MRI achieved the highest sensitivity of 0.952 [0.887–1.000]. In terms of deep learning radiomics, the stacking fusion strategy achieved the highest accuracy, AUC, and sensitivity, with values of 0.937, 0.947 [0.887–1.000], and 1.000 [0.999–1.000], respectively. The early fusion strategies of US+MRI and US+MG achieved the highest specificity of 0.954 [0.867–1.000]. In terms of feature fusion, the ensemble and stacking approaches of the late fusion strategy achieved the highest accuracy of 0.968. In addition, stacking achieved the highest AUC and specificity, which were 0.997 [0.990–1.000] and 1.000 [0.999–1.000], respectively. The traditional radiomic and depth features of US+MG + MR achieved the highest sensitivity of 1.000 [0.999–1.000] under the early fusion strategy. Conclusion This study demonstrated the potential of integrating deep learning and radiomic features with multimodal images. As a single modality, MRI based on radiomic features achieved greater accuracy than US or MG. The US and MG models achieved higher accuracy with transfer learning than the single-mode or radiomic models. The traditional radiomic and depth features of US+MG + MR achieved the highest sensitivity under the early fusion strategy, showed higher diagnostic performance, and provided more valuable information for differentiation between benign and malignant breast tumours.

Radiomics with Attribute Bagging for Breast Tumor Classification Using Multimodal Ultrasound Images

Multi-region radiomics for artificially intelligent diagnosis of breast cancer using multimodal ultrasound

A New Application of Multimodality Radiomics Improves Diagnostic Accuracy of Nonpalpable Breast Lesions in Patients with Microcalcifications-Only in Mammography

Benign and Malignant Classification of Breast Tumor Ultrasound Images Using Conventional Radiomics and Transfer Learning Features: A Multicenter Retrospective Study

Radiomics Analysis on Ultrasound for Prediction of Biologic Behavior in Breast Invasive Ductal Carcinoma

Breast ultrasound tumour classification: A Machine Learning—Radiomics based approach

Breast Tumor Classification based on MRI-US Images by Disentangling Modality Features

Machine Learning-Based Breast Tumor Ultrasound Radiomics for Pre-operative Prediction of Axillary Sentinel Lymph Node Metastasis Burden in Early-Stage Invasive Breast Cancer.

Deep learning radiomics based on multimodal imaging for distinguishing benign and malignant breast tumours

Multi-modality radiomics nomogram based on DCE-MRI and ultrasound images for benign and malignant breast lesion classification

Radiomics Analysis of Breast Lesions in Combination with Coronal Plane of ABVS and Strain Elastography

Ultrasound Radiomics‐Based Logistic Regression Model to Differentiate Between Benign and Malignant Breast Nodules

Constructing the Optimal Classification Model for Benign and Malignant Breast Tumors Based on Multifeature Analysis from Multimodal Images

Multi-Features-Based Automated Breast Tumor Diagnosis Using Ultrasound Image and Support Vector Machine

Multimodal Ultrasound Imaging in Breast Imaging-Reporting and Data System 4 Breast Lesions: A Prediction Model for Malignancy

Breast Tumor Classification in Ultrasound Images by Fusion of Deep Convolutional Neural Network and Shallow LBP Feature

Machine learning radiomics based on intra and peri tumor PA/US images distinguish between luminal and non-luminal tumors in breast cancers

Management of breast lesions seen on US images: dual-model radiomics including shear-wave elastography may match performance of expert radiologists

Ultrasound-based radiomics model for predicting molecular biomarkers in breast cancer

US-based radiomics analysis of different machine learning models for differentiating benign and malignant BI-RADS 4A breast lesions