Performance of readers and an artificial intelligence tool for grading of radiographic knee osteoarthritis at prespecified thresholds: Statistical analysis plan
Mathias Willadsen Brejneboel,Anders Lenskjold,Jacob J. Visser,Huib Ruitenbeek,Katharina Ziegeler,Philip Hansen,Janus Uhd Nybing,Kay Geert A. Hermann,Edwin H.G. Oei,Mikael Boesen
DOI: https://doi.org/10.1101/2024.03.13.24304202
2024-05-27
Abstract:Background and rationale: Knee osteoarthritis (OA) is a common disease characterized by reduced function, stiffness, and pain. This clinical diagnosis is commonly supported with radiography of the weight-bearing knee. Radiographic features, such as the Kellgren-Lawrence (KL) grading system, are used as eligibility criteria for clinical studies while others, such as the OARSI grades and minimal joint space width, are used as endpoints for structural OA progression. A higher preoperative KL-grade has been correlated with better pain- and functional outcomes after knee arthroplasty surgery. Consequently, the KL-grade is a common requirement for approving knee arthroplasty among American health insurance providers and it is commonly used by orthopedic surgeons as part of determining knee arthroplasty candidacy.
Historically, a radiologist was required to draw on and grade radiographs of the knee to extract the features. With increasing computational power and the increased use of deep convolutional neural networks, off-the-shelf artificial intelligence (AI) tools have become available for automatic extraction of these features. They have received regulatory approval for commercialization but it is apparent that more diligent external validation is required. Finally, as AI tools begin to mature, new versions are released. It is important to assess how these developments change the current performance of the tool.
Objectives: The aim of this analysis is to evaluate the performance of a commercially available AI tool and of readers with different experience levels in orthopedic surgery and radiology at clinically relevant Kellgren-Lawrence grading system thresholds.
Methods: This study is a secondary analysis of the data from the AutoRayValid-RBknee study, a retrospective observer performance study. It consists of non-fixed-flexion radiographs acquired from the production picture archiving and communications system (PACS) from three European centers. The primary outcome will be the difference in area under the receiver operating curve (AUC) between the readers and the AI tool at the prior authorization clinical criteria threshold (KL ≥ 3). Key secondary outcomes will be radiographic knee osteoarthritis (KL ≥ 2), osteoarthritis clinical trial inclusion (2 ≤ KL ≤ 3), and weight-loss trial inclusion (1 ≤ KL ≤ 3). The AUC of the readers will be computed using the SROC approach as proposed by Oakden-Rayner et al. Further, the performance of the AI tool for grading ordinal OARSI grades will be evaluated using the ordinal ROC as proposed by Obuchowski et al. and the AUC is used for estimating binary OARSI-grade and patellar osteophyte classification performance.
Population:
Patients with knee pain referred for radiography on suspicion of knee osteoarthritis.
Index test:
Readers: Each center will recruit four readers from across radiology and orthopedic surgery, one in-training and one board-certified for each specialty.
AI tool: RBknee-2.2.0 (CE version, KL-grading, OARSI grading, patellar osteophytes) and RBknee-2.1.0 (CE version, KL-grading, OARSI grading, patellar osteophytes) will be used to perform the change impact analysis of advancing product development.
Reference test:
The reference standard will be determined by the majority vote of three readers, one from each participating hospital who are a board-certified musculoskeletal radiology consultant with expertise in clinical and research evaluation of KOA including extensive experience using the KL-grade.
Further statistical details
Sample size: Not applicable as this is a secondary analysis.
Framework: This is a diagnostic test accuracy study assessing the performance of a commercially available AI tool for radiographic evaluation of knee osteoarthritis according to established grading systems. Additionally, change impact analysis will be performed where multiple versions of the AI tool are available.
Confidence intervals and P values: All 95% confidence intervals and P values will use an alpha of 5%.
Multiplicity: No explicit multiplicity correction will be performed. Instead, a hierarchical approach will be taken based on tabular order of the tested hypotheses in Table 3.
Statistical software: R version 4.2.2 (or newer).