The conditional approach to evaluating detection performance

Wolf Schwarz

DOI: https://doi.org/10.3758/s13414-021-02362-6

Abstract:In many applied single-point Yes/No signal-detection studies, the main interest is to evaluate the observer's sensitivity, based on the observed rates of hits and false alarms. For example, Kostopoulou, Nurek, Cantarella et al. (2019, Medical Decision Making, 39, 21-31) presented general practitioners (GPs) with clinical vignettes of patients showing various cancer-related symptoms, and asked them to decide if urgent referral was required; the standard discrimination index d' was calculated for each GP. An alternative conditional approach to statistical inference emphasizes explicitly the conditional nature of the inferences drawn, and argues on the basis of the response marginal (the number of "yes" responses) that was actually observed. It is closely related to, for example, Fisher's exact test or the Rasch model in item response theory which have long been valuable and prominent in psychology. The conditional framework applied to single-point Yes/No detection studies is based on the noncentral hypergeometric sampling distribution and permits, for samples of any size, exact inference because it eliminates nuisance (i.e., bias) parameters by conditioning. We describe in detail how the conditional approach leads to conditional maximum likelihood sample estimates of sensitivity, and to exact confidence intervals for the underlying (log) odds ratio. We relate the conditional approach to classical (logistic) detection models also leading to analyses of the odds ratio, compare its statistical power to that of the unconditional approach, and conclude by discussing some of its pros and cons.