Abstract:Performance metrics for medical image segmentation models are used to measure the agreement between the reference annotation and the predicted segmentation. Usually, overlap metrics, such as the Dice, are used as a metric to evaluate the performance of these models in order for results to be comparable. However, there is a mismatch between the distributions of cases and difficulty level of segmentation tasks in public data sets compared to clinical practice. Common metrics fail to measure the impact of this mismatch, especially for clinical data sets that include low signal pathologies, a difficult segmentation task, and uncertain, small, or empty reference annotations. This limitation may result in ineffective research of machine learning practitioners in designing and optimizing models. Dimensions of evaluating clinical value include consideration of the uncertainty of reference annotations, independence from reference annotation volume size, and evaluation of classification of empty reference annotations. We study how uncertain, small, and empty reference annotations influence the value of metrics for medical image segmentation on an in-house data set regardless of the model. We examine metrics behavior on the predictions of a standard deep learning framework in order to identify metrics with clinical value. We compare to a public benchmark data set (BraTS 2019) with a high-signal pathology and certain, larger, and no empty reference annotations. We may show machine learning practitioners, how uncertain, small, or empty reference annotations require a rethinking of the evaluation and optimizing procedures. The evaluation code was released to encourage further analysis of this topic. <a class="link-external link-https" href="https://github.com/SophieOstmeier/UncertainSmallEmpty.git" rel="external noopener nofollow">this https URL</a>

Bolstering Performance Evaluation of Image Segmentation Models with Efficacy Metrics in the Absence of a Gold Standard

Troubleshooting image segmentation models with human-in-the-loop

Accuracy Evaluation of Segmentation for High Resolution Imagery and 3D Laser Point Cloud Data

A Novel Quality Evaluating Method for Over-Segmentation Approaches Using Real-Time Boundary Information

Objective Performance Evaluation of Video Segmentation Algorithms with Ground-Truth

Towards a guideline for evaluation metrics in medical image segmentation

Revisiting Evaluation Metrics for Semantic Segmentation: Optimization and Evaluation of Fine-grained Intersection over Union

Multi-Strategy Emperor Penguin Optimizer for RGB Histogram-Based Color Satellite Image Segmentation Using Masi Entropy

A Learning-Based Framework for Image Segmentation Evaluation

New Evaluation Metrics for Mesh Segmentation

Fine-grained Metrics for Point Cloud Semantic Segmentation

Segmentation with Multiple Acceptable Annotations: A Case Study of Myocardial Segmentation in Contrast Echocardiography

A Fast and Effective Method for Unsupervised Segmentation Evaluation of Remote Sensing Images

Performance evaluation of semantic segmentation models: a cross meta-frontier DEA approach

Towards Ground-truth-free Evaluation of Any Segmentation in Medical Images

USE-Evaluator: Performance Metrics for Medical Image Segmentation Models with Uncertain, Small or Empty Reference Annotations

Edge-Based Unsupervised Evaluation of Image Segmentation.

A Weighted-Roc Graph Based Metric for Image Segmentation Evaluation

Experimenting with an Evaluation Framework for Imbalanced Data Learning (EFIDL)

LIME-Eval: Rethinking Low-light Image Enhancement Evaluation via Object Detection

Informing selection of performance metrics for medical image segmentation evaluation using configurable synthetic errors