Methods for analytical validation of novel digital clinical measures: A simulation study

Simon Turner,Chen Chen,Rolando Acosta,Eric J Daza,Lysbeth Floden,Joss Langford,Leif Simmatis,Berend Terluin,Piper Fromy

DOI: https://doi.org/10.1101/2024.11.29.24318211

2024-12-01

Abstract:Analytical validation (AV) is a crucial step in the evaluation of algorithms that process data from sensor-based digital health technologies (sDHTs). AV of novel digital measures can be complicated when reference measures with direct correspondence are not available. To address this, we conducted a simulation study. Data was simulated assuming a latent physical ability trait, indirectly accessed through an sDHT-derived target measure collecting step count data, and the items of a clinical outcome assessment measuring self-reported physical activity. We quantified the ability of two methods to assess the latent relationship between reference and target measures: the Pearson Correlation Coefficient (PCC) (a classical, simple method frequently used for AV), and factor correlations from a two-factor confirmatory factor analysis (CFA) model (a more novel approach to AV, able to capture more complex relationships). Additionally, three multiple linear regression models were used to evaluate if multiple clinical outcome assessment reference measures can more completely represent a target measure of interest. Our findings show that PCC was more stable, easier to compute, and relatively robust with respect to violations of parametric assumptions than CFA, particularly with small sample sizes. However, CFA was less biased than PCC in all scenarios investigated. We demonstrate that using both PCC and CFA allows a researcher to be more confident that their AV results reflect the true relationship between the sDHT-derived target measure and a reference measure. Finally, regression results suggest that incorporating multiple reference measures with more frequent collection time points can provide a more complete presentation of the sDHT’s analytical validity. Novel digital measures are being developed at an accelerating pace and promise to revolutionize public health, patient care, and medical product development. Our findings provide investigators with crucial information for choosing appropriate methods to perform rigorous AV of these novel measures.

Health Informatics