Abstract:Over the past decade, the field of forensic science has received recommendations from the National Research Council of the U.S. National Academy of Sciences, the U.S. National Institute of Standards and Technology, and the U.S. President's Council of Advisors on Science and Technology to study the validity and reliability of forensic analyses. More specifically, these committees recommend estimation of the rates of occurrence of erroneous conclusions drawn from forensic analyses. "Black box" studies for the various subjective feature-based comparison methods are intended for this purpose. In general, "black box" studies often have unbalanced designs, comparisons that are not independent, and missing data. These aspects pose difficulty in the analysis of the results and are often ignored. Instead, interpretation of the data relies on methods that assume independence between observations and a balanced experiment. Furthermore, all of these projects are interpreted within the frequentist framework and result in point estimates associated with confidence intervals that are confusing to communicate and understand. We propose to use an existing likelihood-free Bayesian inference method, called Approximate Bayesian Computation (ABC), that is capable of handling unbalanced designs, dependencies among the observations, and missing data. ABC allows for studying the parameters of interest without recourse to incoherent and misleading measures of uncertainty such as confidence intervals. By taking into account information from all decision categories for a given examiner and information from the population of examiners, our method also allows for quantifying the risk of error for the given examiner, even when no error has been recorded for that examiner. We illustrate our proposed method by reanalysing the results of the "Noblis Black Box" study by Ulery et al. in 2011.

A Note on the Specific Source Identification Problem in Forensic Science in the Presence of Uncertainty about the Background Population

Generalized fiducial factor: an alternative to the Bayes factor for forensic identification of source problems

A Bayesian approach for the analysis of error rate studies in forensic science

Evaluation of distance-based approaches for forensic comparison: Application to hand odor evidence

Forensic identification: the Island Problem and its generalisations

Using Bayes factors to limit forensic testimony to forensics: composite hypotheses

Sensitivity of inferences in forensic genetics to assumptions about founding genes

Defence Against the Modern Arts: the Curse of Statistics "Score-based likelihood ratios"

The Influence of Validation Data on Logical and Scientific Interpretations of Forensic Expert Opinions

Commentary on Guyll et al. (2023): Misuse of Statistical Method Results in Highly Biased Interpretation of Forensic Evidence

Interpreting forensic DNA mixtures: allowing for uncertainty in population substructure and dependence

Estimating probability terms for the background presence of glass when considering activity in forensic casework

Psychometric analysis of forensic examiner behavior

Quantification of the weight of fingerprint evidence using a ROC-based Approximate Bayesian Computation algorithm for model selection

Quantifying the strength of firearms comparisons based on error rate studies

On forensic likelihood ratios from low-coverage sequencing

The Role of the Bayes Factor in the Evaluation of Evidence

Forensic science evidence: Naive estimates of false positive error rates and reliability

Bloodstain pattern analysis & Bayes: A case report

A forensic science perspective on the role of images in crime investigation and reconstruction

Logical implications of applying the principles of population genetics to the interpretation of DNA profiling evidence