Ghada Zamzmi,Kesavan Venkatesh,Brandon Nelson,Smriti Prathapan,Paul H. Yi,Berkman Sahiner,Jana G. Delfino
Abstract:Background: Machine learning (ML) methods often fail with data that deviates from their training distribution. This is a significant concern for ML-enabled devices in clinical settings, where data drift may cause unexpected performance that jeopardizes patient safety.
What problem does this paper attempt to address?
This paper attempts to solve the problem of performance degradation of machine learning (ML) models when dealing with data that deviates from the distribution of their training data, especially in clinical settings, where data drift may lead to unexpected performance changes and thus endanger patient safety. Specifically, the paper focuses on how to detect and monitor "out - of - distribution" (OOD) samples and data drift in medical imaging data.
### Background and Problem
- **Background**: Machine learning methods usually fail when dealing with data that deviates from the distribution of their training data. This is especially important in clinical settings because data drift may lead to an unexpected decline in model performance, which in turn affects patient safety.
- **Problem**: Existing supervised machine learning models perform poorly when dealing with unknown or unrelated inputs. For example, a model used to classify chest X - rays (CXR) may misclassify unrelated knee X - rays. This limitation has led to research in the field of open - world classification, which aims to extend static or closed models to reject unknown images.
### Method
- **Framework**: The authors propose a framework that combines machine learning methods and statistical process control (SPC) for OOD detection and data drift monitoring. The advantage of SPC is that it can highlight deviations from the expected distribution intuitively and statistically.
- **Design Choices**: The paper explores different design choices, including feature representation extraction methods, drift quantification methods, and SPC parameter selection.
### Results
- **Tasks**: The paper demonstrates the effectiveness of this framework in two tasks:
1. Distinguishing between axial CT images and non - axial CT images.
2. Separating chest X - rays (CXR) from images of other modalities.
- **Performance**: For these two tasks, the framework has achieved high accuracy in detecting OOD inputs, with an accuracy of 0.913 for CT images and 0.995 for CXR images, and sensitivities of 0.980 and 0.984 respectively. In addition, the framework can also monitor the data stream and identify when drift occurs. In a simulation experiment, the framework detected an increase in the proportion of OOD inputs from 0 - 1% to 3 - 5% in 100 daily CXR cases and maintained a low false - positive rate.
### Conclusions
- **Contributions**: This is the first time that SPC has been combined with machine learning methods, providing a practical solution for data drift monitoring in medical imaging. The framework is independent of data, modality, and model, and can be customized and adapted to specific applications.
- **Future Directions**: The paper summarizes the research findings and discusses future research directions, including further optimizing feature selection and distance metrics, and exploring more application scenarios.
### Key Technical Details
- **Feature Representation**: The paper explores different feature extraction methods, including unsupervised learning (such as auto - encoders) and supervised learning (such as convolutional neural networks).
- **OOD Metrics**: Geometric distance metric methods are used, including Mahalanobis Distance and Cosine Similarity.
- **SPC Tools**: 3σ control charts and cumulative sum control charts (CUSUM) are used to detect and monitor OOD inputs.
Through these methods, the paper provides a comprehensive framework for detecting and monitoring OOD inputs and data drift in medical imaging, thereby improving the reliability and safety of the model.