Abstract:Background In recent years, there was an increasing popularity in applying artificial intelligence in the medical field from computer-aided diagnosis (CAD) to patient prognosis prediction. Given the fact that not all healthcare professionals have the required expertise to develop a CAD system, the aim of this study was to investigate the feasibility of using AutoML Vision, a highly automatic machine learning model, for future clinical applications by comparing AutoML Vision with some commonly used CAD algorithms in the differentiation of benign and malignant breast lesions on ultrasound. Methods A total of 895 breast ultrasound images were obtained from the two online open-access ultrasound breast images datasets. Traditional machine learning models (comprising of seven commonly used CAD algorithms) with three content-based radiomic features (Hu Moments, Color Histogram, Haralick Texture) extracted, and a convolutional neural network (CNN) model were built using python language. AutoML Vision was trained in Google Cloud Platform. Sensitivity, specificity, F1 score and average precision (AUCPR) were used to evaluate the diagnostic performance of the models. Cochran's Q test was used to evaluate the statistical significance between all studied models and McNemar test was used as the post-hoc test to perform pairwise comparisons. The proposed AutoML model was also compared with the current related studies that involve similar medical imaging modalities in characterizing benign or malignant breast lesions. Results There was significant difference in the diagnostic performance among all studied traditional machine learning classifiers (P<0.05). Random Forest achieved the best performance in the differentiation of benign and malignant breast lesions (accuracy: 90%; sensitivity: 71%; specificity: 100%; F1 score: 0.83; AUCPR: 0.90) which was statistically comparable to the performance of CNN (accuracy: 91%; sensitivity: 82%; specificity: 96%; F1 score: 0.87; AUCPR: 0.88) and AutoML Vision (accuracy: 86%; sensitivity: 84%; specificity: 88%; F1 score: 0.83; AUCPR: 0.95) based on Cochran's Q test (P>0.05). Conclusions In this study, the performance of AutoML Vision was not significantly different from that of Random Forest (the best classifier among traditional machine learning models) and CNN. AutoML Vision showed relatively high accuracy and comparable to current commonly used classifiers which may prompt for future application in clinical practice.

Application Value of Random Forest and Support Vector Machine in Diagno-sing Breast Lesions by Using Ultrasonic Image Features

Prediction model of neck lymphoma based on gray-scale ultrasonography radiomics

Machine Learning Models for Breast Lesions Based on Ultrasound Imaging Features: A Observational Study

Comparison of the diagnostic efficacy of mathematical models in distinguishing ultrasound imaging of breast nodules

Ultrasound Radiomics‐Based Logistic Regression Model to Differentiate Between Benign and Malignant Breast Nodules

Support vector machines based computer-aided diagnosis system of breast tumor with ultrasound images

Evaluation of the performance of traditional machine learning algorithms, convolutional neural network and AutoML Vision in ultrasound breast lesions classification: a comparative study.

Diagnosis of benign and malignant peripheral lung lesions based on a feature model constructed by the random forest algorithm for grayscale and contrast-enhanced ultrasound

Application of Ultrasonographic Score Evaluation System in Differential Diagnosis of Breast Mass Lesions:Comparison with Pathological Data

Influence of the Computer-Aided Decision Support System Design on Ultrasound-Based Breast Cancer Classification

Feature analysis of ultrasound breast tumor images

A Novel Computer-Aided-Diagnosis System for Breast Ultrasound Images Based on BI-RADS Categories

Machine Learning Models to Improve the Differentiation Between Benign and Malignant Breast Lesions on Ultrasound: A Multicenter External Validation Study.

Logistic regression model in evaluation on diagnosis of breast nodules classification of breast imaging reporting and data system 3-5 category with automated breast volume scanner combined with conventional ultrasound

Application of Deep Learning to Establish a Diagnostic Model of Breast Lesions Using Two-Dimensional Grayscale Ultrasound Imaging.

Associated factors leading to misdiagnosis of a combined diagnostic model of different types of strain imaging and conventional ultrasound in evaluation of breast lesions: Selection strategy for using different types of strain imaging in evaluation of breast lesions

Machine Learning Based on Automated Breast Volume Scanner (ABVS) Radiomics for Differential Diagnosis of Benign and Malignant BI‐RADS 4 Lesions

Multimodal Ultrasound Imaging in Breast Imaging-Reporting and Data System 4 Breast Lesions: A Prediction Model for Malignancy

The value of radiomics model based on ultrasound image features in the differentiation between minimal breast cancer and small benign breast masses

Applications of machine-learning algorithms for prediction of benign and malignant breast lesions using ultrasound radiomics signatures: A multi-center study