Abstract:Reliable performance of PET segmentation algorithms on clinically relevant tasks is required for their clinical translation. However, these algorithms are typically evaluated using figures of merit (FoMs) that are not explicitly designed to correlate with clinical task performance. Such FoMs include the Dice similarity coefficient (DSC), the Jaccard similarity coefficient (JSC), and the Hausdorff distance (HD). The objective of this study was to investigate whether evaluating PET segmentation algorithms using these task-agnostic FoMs yields interpretations consistent with evaluation on clinically relevant quantitative tasks. Methods: We conducted a retrospective study to assess the concordance in the evaluation of segmentation algorithms using the DSC, JSC, and HD and on the tasks of estimating the metabolic tumor volume (MTV) and total lesion glycolysis (TLG) of primary tumors from PET images of patients with non–small cell lung cancer. The PET images were collected from the American College of Radiology Imaging Network 6668/Radiation Therapy Oncology Group 0235 multicenter clinical trial data. The study was conducted in 2 contexts: (1) evaluating conventional segmentation algorithms, namely those based on thresholding (SUV max 40% and SUV max 50%), boundary detection (Snakes), and stochastic modeling (Markov random field–Gaussian mixture model); (2) evaluating the impact of network depth and loss function on the performance of a state-of-the-art U-net–based segmentation algorithm. Results: Evaluation of conventional segmentation algorithms based on the DSC, JSC, and HD showed that SUV max 40% significantly outperformed SUV max 50%. However, SUV max 40% yielded lower accuracy on the tasks of estimating MTV and TLG, with a 51% and 54% increase, respectively, in the ensemble normalized bias. Similarly, the Markov random field–Gaussian mixture model significantly outperformed Snakes on the basis of the task-agnostic FoMs but yielded a 24% increased bias in estimated MTV. For the U-net–based algorithm, our evaluation showed that although the network depth did not significantly alter the DSC, JSC, and HD values, a deeper network yielded substantially higher accuracy in the estimated MTV and TLG, with a decreased bias of 91% and 87%, respectively. Additionally, whereas there was no significant difference in the DSC, JSC, and HD values for different loss functions, up to a 73% and 58% difference in the bias of the estimated MTV and TLG, respectively, existed. Conclusion: Evaluation of PET segmentation algorithms using task-agnostic FoMs could yield findings discordant with evaluation on clinically relevant quantitative tasks. This study emphasizes the need for objective task-based evaluation of image segmentation algorithms for quantitative PET.

Segmentation metric misinterpretations in bioimage analysis

Towards a guideline for evaluation metrics in medical image segmentation

Metrics Revolutions: Groundbreaking Insights into the Implementation of Metrics for Biomedical Image Segmentation

Metrics for evaluating 3D medical image segmentation: analysis, selection, and tool

Pitfalls of topology-aware image segmentation

What Matters in Radiological Image Segmentation? Effect of Segmentation Errors on the Diagnostic Related Features

A framework for evaluating image segmentation algorithms

Need for Objective Task-Based Evaluation of Image Segmentation Algorithms for Quantitative PET: A Study with ACRIN 6668/RTOG 0235 Multicenter Clinical Trial Data

Misclassified Evaluation Metrics for 3D Mesh Segmentation

Toward Objective Evaluation of Image Segmentation Algorithms

Objective Performance Evaluation of Video Segmentation Algorithms with Ground-Truth

A Weighted-Roc Graph Based Metric for Image Segmentation Evaluation

Optimizing the Dice Score and Jaccard Index for Medical Image Segmentation: Theory & Practice

Optimization for Medical Image Segmentation: Theory and Practice when evaluating with Dice Score or Jaccard Index

Informing selection of performance metrics for medical image segmentation evaluation using configurable synthetic errors

Segmentation Quality and Volumetric Accuracy in Medical Imaging

Analyzing Image Segmentation for Connectomics

Objective Image Quality Measures and Their Applications in Segmentation Evaluation

Understanding the Impact of Evaluation Metrics in Kinetic Models for Consensus-based Segmentation

Evaluation and benchmark for biological image segmentation

The Medical Segmentation Decathlon