Abstract:There is no consensus regarding the radiomic feature terminology, the underlying mathematics, or their implementation. This creates a scenario where features extracted using different toolboxes could not be used to build or validate the same model leading to a non-generalization of radiomic results. In this study, the image biomarker standardization initiative (IBSI) established phantom and benchmark values were used to compare the variation of the radiomic features while using 6 publicly available software programs and 1 in-house radiomics pipeline. All IBSI-standardized features (11 classes, 173 in total) were extracted. The relative differences between the extracted feature values from the different software and the IBSI benchmark values were calculated to measure the inter-software agreement. To better understand the variations, features are further grouped into 3 categories according to their properties: 1) morphology, 2) statistic/histogram and 3)texture features. While a good agreement was observed for a majority of radiomics features across the various programs, relatively poor agreement was observed for morphology features. Significant differences were also found in programs that use different gray level discretization approaches. Since these programs do not include all IBSI features, the level of quantitative assessment for each category was analyzed using Venn and the UpSet diagrams and also quantified using two ad hoc metrics. Morphology features earns lowest scores for both metrics, indicating that morphological features are not consistently evaluated among software programs. We conclude that radiomic features calculated using different software programs may not be identical and reliable. Further studies are needed to standardize the workflow of radiomic feature extraction.

What problem does this paper attempt to address?

The main problem that this paper attempts to solve is the inconsistency of radiomics features among different software tools. Specifically: 1. **Lack of consensus in terms, mathematical definitions and implementations**: Currently, different radiomics toolkits have no unified standards for feature terms, mathematical definitions and their implementation methods. This results in the features extracted using different tools being unable to be used to construct or validate the same model, thus affecting the generalization ability of radiomics results. 2. **Differences in feature calculations**: In the study, by using the digital phantom and benchmark values established by the Image Biomarker Standardization Initiative (IBSI), the differences among 6 publicly available software programs and 1 in - house radiomics pipeline in extracting 173 IBSI - standardized features were compared. The results showed that although most radiomics features showed good consistency among different programs, the consistency of morphological features was poor. 3. **Impact of gray - level discretization methods**: The study also found that there were significant differences among programs using different gray - level discretization methods. These differences may stem from problems such as intensity shift in the pre - processing steps. 4. **Differences among feature categories**: To better understand these differences, features were divided into three categories: morphological features, statistical/histogram features and texture features. Morphological features performed the worst, indicating that these features were not evaluated consistently enough in different software. 5. **Need for standardization**: The paper points out that since radiomics features calculated by different software programs may not be exactly the same and are unreliable, further research is needed to standardize the radiomics feature extraction workflow. ### Formula summary - **Gray - level discretization formula**: - Fixed bin number: \[ X_{d,k}=\begin{cases}\left\lfloor\frac{N_g(X_{gl,k}-X_{gl,\min})}{X_{gl,\max}-X_{gl,\min}}\right\rfloor + 1&\text{if }X_{gl,k}<X_{gl,\max}\\N_g&\text{if }X_{gl,k}=X_{gl,\max}\end{cases} \] - Fixed bin width: \[ X_{d,k}=\left\lfloor\frac{X_{gl,k}-X_{gl,\min}}{w_b}\right\rfloor + 1 \] - **Relative difference formula**: \[ \text{relative difference}=\frac{\vert\text{feature value}-\text{benchmark value}\vert}{\text{benchmark value}} \] - **Popularity index formula**: - Popularity 1: \[ P_1=\frac{\sum_{i = 1}^{d}w_i}{6d},\quad0\leq P_1\leq1 \] - Popularity 2: \[ P_2=\frac{\sum_{i = 1}^{d}1(w_i>4)}{d},\quad0\leq P_2\leq1 \] ### Conclusion This study emphasizes the consistency and differences in feature calculations among different radiomics software and points out the problems existing in the calculation of morphological features. To improve the reliability and reproducibility of radiomics analysis, future research should be dedicated to standardizing the feature extraction workflow and ensuring consistency among different software.

Benchmarking features from different radiomics toolkits / toolboxes using Image Biomarkers Standardization Initiative

CT and MRI radiomic features of lung cancer (NSCLC): comparison and software consistency

The Image Biomarker Standardization Initiative: Standardized Convolutional Filters for Reproducible Radiomics and Enhanced Clinical Insights

A heterogeneous phantom study for investigating the stability of PET images radiomic features with varying reconstruction settings

Radiomics reproducibility in computed tomography through changes of ROI size, resolution, and hounsfield unit: A phantom study

Cancer radiomic feature variations due to reconstruction kernel choice and integral tube current.

Reproducibility and Repeatability of CBCT-Derived Radiomics Features

Radiomics as a measure superior to common similarity metrics for tumor segmentation performance evaluation

Resampling and harmonization for mitigation of heterogeneity in image parameters of baseline scans

Enhancing the Clinical Utility of Radiomics: Addressing the Challenges of Repeatability and Reproducibility in CT and MRI

Open-radiomics: A Collection of Standardized Datasets and a Technical Protocol for Reproducible Radiomics Machine Learning Pipelines

Understanding Sources of Variation to Improve the Reproducibility of Radiomics

Myocardial perfusion SPECT radiomic features reproducibility assessment: Impact of image reconstruction and harmonization

Influence of inter-observer delineation variability on radiomics stability in different tumor sites

Reproducibility of quantitative high-throughput BI-RADS features extracted from ultrasound images of breast cancer.

Radiomics feature reproducibility under inter-rater variability in segmentations of CT images

Radiomics as a measure superior to the Dice similarity coefficient for tumor segmentation performance evaluation

Repeatability and Reproducibility of Computed Tomography Radiomics for Pulmonary Nodules A Multicenter Phantom Study

Robustness of radiomics among photon-counting detector CT and dual-energy CT systems: a texture phantom study

Repeatability and Reproducibility of Computed Tomography Radiomics for Pulmonary Nodules

Multiparametric Data-driven Imaging Markers: Guidelines for Development, Application and Reporting of Model Outputs in Radiomics