Abstract:Purpose: To (1) analyze trends in the publishing of statistical fragility index (FI)-based systematic reviews in the orthopaedic literature, including the prevalence of misleading or inaccurate statements related to the statistical fragility of randomized controlled trials (RCTs) and patients lost to follow-up (LTF), and (2) determine whether RCTs with relatively "low" FIs are truly as sensitive to patients LTF as previously portrayed in the literature. Methods: All FI-based studies published in the orthopaedic literature were identified using the Cochrane Database of Systematic Reviews, Web of Science Core Collection, PubMed, and MEDLINE databases. All articles involving application of the FI or reverse FI to study the statistical fragility of studies in orthopaedics were eligible for inclusion in the study. Study characteristics, median FIs and sample sizes, and misleading or inaccurate statements related to the FI and patients LTF were recorded. Misleading or inaccurate statements-defined as those basing conclusions of trial fragility on the false assumption that adding patients LTF back to a trial has the same statistical effect as existing patients in a trial experiencing the opposite outcome-were determined by 2 authors. A theoretical RCT with a sample size of 100, P = .006, and FI of 4 was used to evaluate the difference in effect on statistical significance between flipping outcome events of patients already included in the trial (FI) and adding patients LTF back to the trial to show the true sensitivity of RCTs to patients LTF. Results: Of the 39 FI-based studies, 37 (95%) directly compared the FI with the number of patients LTF. Of these 37 studies, 22 (59%) included a statement regarding the FI and patients LTF that was determined to be inaccurate or misleading. In the theoretical RCT, a reversal of significance was not observed until 7 patients LTF (nearly twice the FI) were added to the trial in the distribution of maximal significance reversal. Conclusions: The claim that any RCT in which the number of patients LTF exceeds the FI could potentially have its significance reversed simply by maintaining study follow-ups is commonly inaccurate and prevalent in orthopaedic studies applying the FI. Patients LTF and the FI are not equivalent. The minimum number of patients LTF required to flip the significance of a typical RCT was shown to be greater than the FI, suggesting that RCTs with relatively low FIs may not be as sensitive to patients LTF as previously portrayed in the literature; however, only a holistic approach that considers the context in which the trial was conducted, potential biases, and study results can determine the merits of any particular RCT. Clinical relevance: Surgeons may benefit from re-examining their interpretation of prior FI reviews that have made claims of substantial RCT fragility based on comparisons between the FI and patients LTF; it is possible the results are more robust than previously believed.

LFK index does not reliably detect small‐study effects in meta‐analysis: A simulation study

Testing for Publication Bias in Diagnostic Meta-Analysis: A Simulation Study

Examining publication bias—a simulation-based evaluation of statistical tests on publication bias

Bias in meta-analysis detected by a simple, graphical test

Funnel plots may show asymmetry in the absence of publication bias with continuous outcomes dependent on baseline risk: presentation of a new publication bias test

Operating Characteristics of a Rank Correlation Test for Publication Bias

Empirical Comparison of Publication Bias Tests in Meta-Analysis

Hybrid test for publication bias in meta-analysis

Hartung-Knapp-Sidik-Jonkman approach and its modification for random-effects meta-analysis with few studies

Improving the error rates of the Begg and Mazumdar test for publication bias in fixed effects meta-analysis

Small Studies May Overestimate the Effect Sizes in Critical Care Meta-Analyses: A Meta-Epidemiological Study

Relative likelihood ratios for neutral comparisons of statistical tests in simulation studies

A modified regression method to test publication bias in meta-analyses with binary outcomes

Systematic Evaluation and Comparison of Statistical Tests for Publication Bias

The Dilemma of Heterogeneity Tests in Meta-Analysis: A Challenge from a Simulation Study

Differential losses to follow-up that are outcome-dependent can vitiate a clinical trial: Simulation results

Investigation of publication bias in meta-analyses of diagnostic test accuracy: a meta-epidemiological study

A re-analysis of about 60,000 sparse data meta-analyses suggests that using an adequate method for pooling matters

The problem of natural funnel asymmetries: a simulation analysis of meta-analysis in macroeconomics

The Number of Patients Lost to Follow-Up May Exceed the Fragility Index of a Randomized Controlled Trial Without Reversing Statistical Significance: A Systematic Review and Statistical Model

Publication Bias in Meta-Analysis: Confidence Intervals for Rosenthal's Fail-Safe Number