Abstract:The likelihood ratio paradigm for quantifying the strength of evidence has been researched in many fields of forensic science. Within this paradigm, score-based approaches for estimating likelihood ratios are becoming more prevalent in the forensic science literature. In this study, a score-based approach for estimating likelihood ratios is implemented for linguistic text evidence. Text data are represented via a bag-of-words model with the Z-score normalised relative frequencies of selected most-frequent words (the number of the most-frequent words = N), and the Euclidean, Manhattan and Cosine distance measures are trialled as the score-generating functions for comparing paired text samples. The score-to-likelihood-ratio conversion model was built using a common source method, and the best fitting model was selected from the parametric models of the Normal, Log-normal, Gamma and Weibull distributions. With the Amazon Product Data Authorship Verification Corpus, two groups of documents (each group including documents of approximately 700, 1400 and 2100 words) were synthesised for each author, allowing 720 same-author comparisons and 517,680 different-author comparisons to test the validity of the system. A series of experiments was conducted using combinations of the following conditions: the three score functions, the different values of N for the feature vector and the different document lengths. The validity of the system was assessed using the log-likelihood-ratio cost (Cllr), and the strength of the derived likelihood ratios was charted in the form of Tippett plots. It was demonstrated that 1) the Cosine measure consistently outperforms the other measures-the best performance is achieved with N = 260, regardless of the document length (e.g., Cllr values of 0.70640, 0.45314 and 0.30692, respectively, for 700, 1400 and 2100 words)-and 2) the derived likelihood ratios are very well calibrated irrespective of the distance measures and document lengths. A follow-up experiment showed that the described score-based approach is relatively robust and stable for a limited quantity of background data. The derived likelihood ratios that were estimated separately to the three distance measures were logistic regression fused; and the fusion achieved a further improvement in performance-for example, a Cllr of 0.23494 for 2100 words. This study demonstrates the possibility of designing likelihood ratio-based systems that discriminate between same-author and different-author documents.

Defence Against the Modern Arts: the Curse of Statistics "Score-based likelihood ratios"

Calculation of forensic likelihood ratios: Use of Monte Carlo simulations to compare the output of score-based approaches with true likelihood-ratio values

A Bayesian approach for the analysis of error rate studies in forensic science

Likelihood Ratio as Weight of Forensic Evidence: A Metrological Perspective

Implementing statistical learning methods through Bayesian networks. Part 1: a guide to Bayesian parameter estimation using forensic science data

Using Bayes factors to limit forensic testimony to forensics: composite hypotheses

Quantifying the strength of firearms comparisons based on error rate studies

The Role of the Bayes Factor in the Evaluation of Evidence

The Influence of Validation Data on Logical and Scientific Interpretations of Forensic Expert Opinions

Demonstrative Evidence and the Use of Algorithms in Jury Trials

Score-based likelihood ratios for linguistic text evidence with a bag-of-words model

Likelihood Ratios for physical traits in forensic investigations

Subtleties in Bayesian decision-theoretic analysis for forensic findings: Notes on recent discussion of the role of validation study data in rational decision making

Psychometric analysis of forensic examiner behavior

Quantification of the weight of fingerprint evidence using a ROC-based Approximate Bayesian Computation algorithm for model selection

Reconciling the Bayes Factor and Likelihood Ratio for Two Non-Nested Model Selection Problems

Bayesian Reasoning and Evidence Communication

A review of likelihood ratios in forensic science based on a critique of Stiffelman "No longer the Gold standard: Probabilistic genotyping is changing the nature of DNA evidence in criminal trials"

Calculating the Likelihood Ratio for Multiple Pieces of Evidence

Commentary on Guyll et al. (2023): Misuse of Statistical Method Results in Highly Biased Interpretation of Forensic Evidence

Towards a Likelihood Ratio Approach for Bloodstain Pattern Analysis