Abstract:Although deep learning models for abnormality classification can perform well in screening mammography, the demographic, imaging, and clinical characteristics associated with increased risk of model failure remain unclear. This retrospective study uses the Emory BrEast Imaging Dataset(EMBED) containing mammograms from 115931 patients imaged at Emory Healthcare between 2013-2020, with BI-RADS assessment, region of interest coordinates for abnormalities, imaging features, pathologic outcomes, and patient demographics. Multiple deep learning models were trained to distinguish between abnormal tissue patches and randomly selected normal tissue patches from screening mammograms. We assessed model performance by subgroups defined by age, race, pathologic outcome, tissue density, and imaging characteristics and investigated their associations with false negatives (FN) and false positives (FP). We also performed multivariate logistic regression to control for confounding between subgroups. The top-performing model, ResNet152V2, achieved accuracy of 92.6%(95%CI=92.0-93.2%), and AUC 0.975(95%CI=0.972-0.978). Before controlling for confounding, nearly all subgroups showed statistically significant differences in model performance. However, after controlling for confounding, we found lower FN risk associates with Other race(RR=0.828;p=.050), biopsy-proven benign lesions(RR=0.927;p=.011), and mass(RR=0.921;p=.010) or asymmetry(RR=0.854;p=.040); higher FN risk associates with architectural distortion (RR=1.037;p<.001). Higher FP risk associates to BI-RADS density C(RR=1.891;p<.001) and D(RR=2.486;p<.001). Our results demonstrate subgroup analysis is important in mammogram classifier performance evaluation, and controlling for confounding between subgroups elucidates the true associations between variables and model failure. These results can help guide developing future breast cancer detection models.

Permutation Test for Image‐on‐Scalar Regression With an Application to Breast Cancer

Use of Response Permutation to Measure an Imaging Dataset's Susceptibility to Overfitting by Selected Standard Analysis Pipelines

Statistical Analysis of Quantitative Cancer Imaging Data

Modeling correlated pairs of mammogram images

Algorithmic encoding of protected characteristics in image-based models for disease detection

Using Machine Learning to Automate Mammogram Images Analysis

Decoupling Inherent Risk and Early Cancer Signs in Image-based Breast Cancer Risk Models

Heterogeneous Image-based Classification Using Distributional Data Analysis

A PERMUTATION TEST FOR TWO-SAMPLE MEANS AND SIGNAL IDENTIFICATION OF HIGH-DIMENSIONAL DATA

Generalized Permutation Framework for Testing Model Variable Significance

Unsupervised Clustering of Quantitative Imaging Phenotypes using Autoencoder and Gaussian Mixture Model

Regression Analyses of Distributions using Quantile Functional Regression

Multivariate quantile-based permutation tests with application to functional data

A Survey of Breast Cancer Screening Techniques: Thermography and Electrical Impedance Tomography

Breast Cancer Diagnosis Using Machine Learning Techniques

Artificial Intelligence for Image-Based Breast Cancer Risk Prediction Using Attention

Statistical Methods for Comparing Predictive Values in Medical Diagnosis

A multi-model approach integrating whole-slide imaging and clinicopathologic features to predict breast cancer recurrence risk

Spatial process decomposition for quantitative imaging biomarkers using multiple images of varying shapes

Prediction of Breast Cancer Recurrence Risk Using a Multi-Model Approach Integrating Whole Slide Imaging and Clinicopathologic Features

Multivariate Analysis on Performance Gaps of Artificial Intelligence Models in Screening Mammography