Abstract:Over the past decade, the field of forensic science has received recommendations from the National Research Council of the U.S. National Academy of Sciences, the U.S. National Institute of Standards and Technology, and the U.S. President's Council of Advisors on Science and Technology to study the validity and reliability of forensic analyses. More specifically, these committees recommend estimation of the rates of occurrence of erroneous conclusions drawn from forensic analyses. "Black box" studies for the various subjective feature-based comparison methods are intended for this purpose. In general, "black box" studies often have unbalanced designs, comparisons that are not independent, and missing data. These aspects pose difficulty in the analysis of the results and are often ignored. Instead, interpretation of the data relies on methods that assume independence between observations and a balanced experiment. Furthermore, all of these projects are interpreted within the frequentist framework and result in point estimates associated with confidence intervals that are confusing to communicate and understand. We propose to use an existing likelihood-free Bayesian inference method, called Approximate Bayesian Computation (ABC), that is capable of handling unbalanced designs, dependencies among the observations, and missing data. ABC allows for studying the parameters of interest without recourse to incoherent and misleading measures of uncertainty such as confidence intervals. By taking into account information from all decision categories for a given examiner and information from the population of examiners, our method also allows for quantifying the risk of error for the given examiner, even when no error has been recorded for that examiner. We illustrate our proposed method by reanalysing the results of the "Noblis Black Box" study by Ulery et al. in 2011.

Reconciling the Bayes Factor and Likelihood Ratio for Two Non-Nested Model Selection Problems

Defence Against the Modern Arts: the Curse of Statistics "Score-based likelihood ratios"

Evaluating the Relative Merits of Competing Models Based on Empirical Likelihood Ratio Test

The flaw of averages: Bayes factors as posterior means of the likelihood ratio

Bayesian Model Selection Methods and Their Application to Biological ODE Systems

Calculating the Likelihood Ratio for Multiple Pieces of Evidence

Nonparametric Bayes Factors Based on Empirical Likelihood Ratios.

Bayesian Model Selection in High-Dimensional Settings

Bayesian Restricted Likelihood Methods: Conditioning on Insufficient Statistics in Bayesian Regression

Quantification of the weight of fingerprint evidence using a ROC-based Approximate Bayesian Computation algorithm for model selection

Calculation of forensic likelihood ratios: Use of Monte Carlo simulations to compare the output of score-based approaches with true likelihood-ratio values

A Bayesian approach for the analysis of error rate studies in forensic science

Resolving conflicts between statistical methods by probability combination: Application to empirical Bayes analyses of genomic data

A frequentist two-sample test based on Bayesian model selection

A puzzle of proportions: Two popular Bayesian tests can yield dramatically different conclusions

Marginal Likelihood Computation for Model Selection and Hypothesis Testing: An Extensive Review

Joining Forces of Bayesian and Frequentist Methodology: A Study for Inference in the Presence of Non-Identifiability

Bayesian Analysis of Non-Linear Structural Equation Models with Non-Ignorable Missing Outcomes from Reproductive Dispersion Models.

Model robust hybrid likelihood

Assessing Bayes Factor Surfaces Using Interactive Visualization and Computer Surrogate Modeling

The Role of the Bayes Factor in the Evaluation of Evidence