Abstract:Statistical Methods in Medical Research, Ahead of Print. Diagnostic accuracy studies assess the sensitivity and specificity of a new index test in relation to an established comparator or the reference standard. The development and selection of the index test are usually assumed to be conducted prior to the accuracy study. In practice, this is often violated, for instance, if the choice of the (apparently) best biomarker, model or cutpoint is based on the same data that is used later for validation purposes. In this work, we investigate several multiple comparison procedures which provide family-wise error rate control for the emerging multiple testing problem. Due to the nature of the co-primary hypothesis problem, conventional approaches for multiplicity adjustment are too conservative for the specific problem and thus need to be adapted. In an extensive simulation study, five multiple comparison procedures are compared with regard to statistical error rates in least-favourable and realistic scenarios. This covers parametric and non-parametric methods and one Bayesian approach. All methods have been implemented in the new open-source R package cases which allows us to reproduce all simulation results. Based on our numerical results, we conclude that the parametric approaches (maxT and Bonferroni) are easy to apply but can have inflated type I error rates for small sample sizes. The two investigated Bootstrap procedures, in particular the so-called pairs Bootstrap, allow for a family-wise error rate control in finite samples and in addition have a competitive statistical power.

Simultaneous confidence intervals for pairwise multiple comparisons in a two-way unbalanced design with unequal variances

Multiple Comparisons of Treatment Against Control under Unequal Variances Using Parametric Bootstrap

A Comparative Review of Methods for Comparing Means Using Partially Paired Data

A Direct Approach to Simultaneous Tests of Superiority and Noninferiority with Multiple Endpoints

A Parametric Bootstrap Approach for One-Way ANOVA Under Unequal Variances with Unbalanced Data.

Multiple Comparisons of Several Log-normal Means under Heteroscedasticity

Parametric Bootstrap Procedures for Three-Factor ANOVA and Multiple Comparison Procedures with Unequal Group Variances

An optimal exact confidence interval for the difference of two independent binomial proportions

An Improved Dunnett's Procedure for Comparing Multiple Treatments with a Control in the Presence of Missing Observations

A Practical Approach for Comparing Means of Two Groups Without Equal Variance Assumption

Confidence intervals for overall response rate difference in the sequential parallel comparison design

Statistical inference for diagnostic test accuracy studies with multiple comparisons

Bootstrap tests for simultaneous monotone ordering of effects in a two‐way ANOVA

Simultaneous Confidence Bands for Comparing Variance Functions of Two Samples Based on Deterministic Designs

Comparison of Software Algorithms for Calculating Reml Wald Type Confidence Limits for the Between-Group Variance Component in A Small Sample One-Way Random Effects Model Example

Simultaneous Confidence Intervals for Several Inverse Gaussian Populations

Simultaneous confidence interval construction for many-to-one of proportion ratios of bilateral correlated data

Common odds ratio test and interval estimation for stratified bilateral and unilateral data

Cost-Efficient Fixed-Width Confidence Intervals for the Difference of Two Bernoulli Proportions

Unconditional Exact Tests for Equivalence or Noninferiority for Paired Binary Endpoints

Hypothesis Test to Compare Two Paired Binomial Proportions: Assessment of 24 Methods