Abstract:Over the past decade, the field of forensic science has received recommendations from the National Research Council of the U.S. National Academy of Sciences, the U.S. National Institute of Standards and Technology, and the U.S. President's Council of Advisors on Science and Technology to study the validity and reliability of forensic analyses. More specifically, these committees recommend estimation of the rates of occurrence of erroneous conclusions drawn from forensic analyses. "Black box" studies for the various subjective feature-based comparison methods are intended for this purpose. In general, "black box" studies often have unbalanced designs, comparisons that are not independent, and missing data. These aspects pose difficulty in the analysis of the results and are often ignored. Instead, interpretation of the data relies on methods that assume independence between observations and a balanced experiment. Furthermore, all of these projects are interpreted within the frequentist framework and result in point estimates associated with confidence intervals that are confusing to communicate and understand. We propose to use an existing likelihood-free Bayesian inference method, called Approximate Bayesian Computation (ABC), that is capable of handling unbalanced designs, dependencies among the observations, and missing data. ABC allows for studying the parameters of interest without recourse to incoherent and misleading measures of uncertainty such as confidence intervals. By taking into account information from all decision categories for a given examiner and information from the population of examiners, our method also allows for quantifying the risk of error for the given examiner, even when no error has been recorded for that examiner. We illustrate our proposed method by reanalysing the results of the "Noblis Black Box" study by Ulery et al. in 2011.

A Unified Framework For Performance Analysis Of Bayesian Inference

A Unified Bayesian Inference Framework for Generalized Linear Models

Analysis of the performance and sensitivity of eigendecomposition-based detectors

An Efficient Variational Bayesian Inference Approach Via Studient's-t Priors for Acoustic Imaging in Colored Noises

Robust CFAR Detection in Heterogeneous Weibull Background Via Bayesian Area Interference Control

A Bayesian Framework of Non-Synchronous Measurements at Coprime Positions for Sound Source Localization with High Resolution

Order Parameters for Detecting Target Curves in Images: When Does High Level Knowledge Help?

Optimal ROC-Based Classification and Performance Analysis under Bayesian Uncertainty Models

A Bayesian approach for the analysis of error rate studies in forensic science

A Unified Bayesian Framework for Modeling Measurement Error in Multinomial Data

Prediction of the Performance of an Algorithm for the Detection of Small Targets in Infrared Images

Bayesian inference in physics

Bayesian inference in the framework of uncertainty theory

Bayesian Bandit Algorithms with Approximate Inference in Stochastic Linear Bandits

Performance evaluation metrics for Track-Before-Detect algorithms

Bayesian Performance Comparison of Text Classifiers

Optimization and Analysis of PDAF with Bayesian Detection.

Mesoscopic Bayesian Inference by Solvable Models

Offline Performance Prediction of PDAF With Bayesian Detection for Tracking in Clutter

A Bayesian Framework for Joint Target Tracking, Classification, and Intent Inference

Performance Analysis Method of Extended Target Detection under Correlated K-distributed Clutter Background