Abstract:Reliable performance of PET segmentation algorithms on clinically relevant tasks is required for their clinical translation. However, these algorithms are typically evaluated using figures of merit (FoMs) that are not explicitly designed to correlate with clinical task performance. Such FoMs include the Dice similarity coefficient (DSC), the Jaccard similarity coefficient (JSC), and the Hausdorff distance (HD). The objective of this study was to investigate whether evaluating PET segmentation algorithms using these task-agnostic FoMs yields interpretations consistent with evaluation on clinically relevant quantitative tasks. Methods: We conducted a retrospective study to assess the concordance in the evaluation of segmentation algorithms using the DSC, JSC, and HD and on the tasks of estimating the metabolic tumor volume (MTV) and total lesion glycolysis (TLG) of primary tumors from PET images of patients with non–small cell lung cancer. The PET images were collected from the American College of Radiology Imaging Network 6668/Radiation Therapy Oncology Group 0235 multicenter clinical trial data. The study was conducted in 2 contexts: (1) evaluating conventional segmentation algorithms, namely those based on thresholding (SUV max 40% and SUV max 50%), boundary detection (Snakes), and stochastic modeling (Markov random field–Gaussian mixture model); (2) evaluating the impact of network depth and loss function on the performance of a state-of-the-art U-net–based segmentation algorithm. Results: Evaluation of conventional segmentation algorithms based on the DSC, JSC, and HD showed that SUV max 40% significantly outperformed SUV max 50%. However, SUV max 40% yielded lower accuracy on the tasks of estimating MTV and TLG, with a 51% and 54% increase, respectively, in the ensemble normalized bias. Similarly, the Markov random field–Gaussian mixture model significantly outperformed Snakes on the basis of the task-agnostic FoMs but yielded a 24% increased bias in estimated MTV. For the U-net–based algorithm, our evaluation showed that although the network depth did not significantly alter the DSC, JSC, and HD values, a deeper network yielded substantially higher accuracy in the estimated MTV and TLG, with a decreased bias of 91% and 87%, respectively. Additionally, whereas there was no significant difference in the DSC, JSC, and HD values for different loss functions, up to a 73% and 58% difference in the bias of the estimated MTV and TLG, respectively, existed. Conclusion: Evaluation of PET segmentation algorithms using task-agnostic FoMs could yield findings discordant with evaluation on clinically relevant quantitative tasks. This study emphasizes the need for objective task-based evaluation of image segmentation algorithms for quantitative PET.

Validation of SynthSeg segmentation performance on CT using paired MRI from radiotherapy patients

Deep learning-based brain segmentation model performance validation with clinical radiotherapy CT

Robust machine learning segmentation for large-scale analysis of heterogeneous clinical brain MRI datasets

Robust Segmentation of Brain MRI in the Wild with Hierarchical CNNs and no Retraining

Region of interest focused MRI to synthetic CT translation using regression and segmentation multi-task network

SynthSeg: Segmentation of brain MRI scans of any contrast and resolution without retraining

CycleSeg: Simultaneous synthetic CT generation and unsupervised segmentation for MR‐only radiotherapy treatment planning of prostate cancer

The Segment Anything foundation model achieves favorable brain tumor autosegmentation accuracy on MRI to support radiotherapy treatment planning

Improving segmentation reliability of multi-scanner brain images using a generative adversarial network

SynSeg-Net: Synthetic Segmentation Without Target Modality Ground Truth

Development and validation of a deep learning-based automatic segmentation model for assessing intracranial volume: comparison with NeuroQuant, FreeSurfer, and SynthSeg

Deep Learning-Based Synthetic Computed Tomography for Low-Field Brain Magnetic Resonance-Guided Radiation Therapy

SynthSR: A public AI tool to turn heterogeneous clinical brain scans into high-resolution T1-weighted images for 3D morphometry

Clinical capability of modern brain tumor segmentation models

Initial Study On Improving Segmentation By Combining Preoperative CT And Intraoperative CBCT Using Synthetic Data

Achieving accurate prostate auto-segmentation on CT in the absence of MR imaging

Performance analysis and knowledge-based quality assurance of critical organ auto-segmentation for pediatric craniospinal irradiation

Comparative assessment of established and deep learning‐based segmentation methods for hippocampal volume estimation in brain magnetic resonance imaging analysis

Improvement of Multiparametric MR Image Segmentation by Augmenting the Data With Generative Adversarial Networks for Glioma Patients

Need for Objective Task-Based Evaluation of Image Segmentation Algorithms for Quantitative PET: A Study with ACRIN 6668/RTOG 0235 Multicenter Clinical Trial Data

Shape-constrained deformable brain segmentation: Methods and quantitative validation