Abstract:Background Among observational studies of routinely collected healthcare data (RCD) for exploring treatment effects, algorithms are used to identify study variables. However, the extent to which algorithms are reliable and impact the credibility of effect estimates is far from clear. This study aimed to investigate the validation of algorithms for identifying study variables from RCD, and examine the impact of alternative algorithms on treatment effects. Methods We searched PubMed for observational studies published in 2018 that used RCD to explore drug treatment effects. Information regarding the reporting, validation, and interpretation of algorithms was extracted. We summarized the reporting and methodological characteristics of algorithms and validation. We also assessed the divergence in effect estimates given alternative algorithms by calculating the ratio of estimates of the primary versus alternative analyses. Results A total of 222 studies were included, of which 93 (41.9%) provided a complete list of algorithms for identifying participants, 36 (16.2%) for exposure, and 132 (59.5%) for outcomes, and 15 (6.8%) for all study variables including population, exposure, and outcomes. Fifty-nine (26.6%) studies stated that the algorithms were validated, and 54 (24.3%) studies reported methodological characteristics of 66 validations, among which 61 validations in 49 studies were from the cross-referenced validation studies. Of those 66 validations,22 (33.3%) reported sensitivity and 16 (24.2%) reported specificity. A total of 63.6% of studies reporting sensitivity and 56.3% reporting specificity used test-result-based sampling, an approach that potentially biases effect estimates. Twenty-eight (12.6%) studies used alternative algorithms to identify study variables, and 24 reported the effects estimated by primary analyses and sensitivity analyses. Of these, 20% had differential effect estimates when using alternative algorithms for identifying population, 18.2% for identifying exposure, and 45.5% for classifying outcomes. Only 32 (14.4%) studies discussed how the algorithms may affect treatment estimates. Conclusions In observational studies of RCD, the algorithms for variable identification were not regularly validated, and – even if validated – the methodological approach and performance of the validation were often poor. More seriously, different algorithms may yield differential treatment effects, but their impact is often ignored by researchers. Strong efforts, including recommendations, are warranted to improve good practice.

Reporting, Handling, and Interpretation of Time-Varying Drug Treatments in Observational Studies Using Routinely Collected Healthcare Data.

Handling time‐varying treatments in observational studies: A scoping review and recommendations

[Methods for Controlling Time-Varying Confounding in Pharmaco-Epidemiological Studies: a Systematic Reveiw].

Evaluation of adaptive treatment strategies in an observational study where time-varying covariates are not monitored systematically

Estimating individualized treatment rules in longitudinal studies with covariate-driven observation times

Missing data were poorly reported and handled in randomized controlled trials with repeatedly measured continuous outcomes: a cross-sectional survey

Validation and impact of algorithms for identifying variables in observational studies of routinely collected data

The reporting quality of randomized controlled trials in pharmacotherapy for pituitary adenomas

Impact of the Consolidated Standards of Reporting Trials (CONSORT) checklist on reporting of randomized clinical trials in traditional Chinese medicine

Assessment of the Reporting Quality of Randomized Controlled Trials on Treatment of Coronary Heart Disease with Traditional Chinese Medicine from the Chinese Journal of Integrated Traditional and Western Medicine: A Systematic Review

Assessment of the Reporting Quality of Placebo-controlled Randomized Trials on the Treatment of Type 2 Diabetes With Traditional Chinese Medicine in Mainland China: A PRISMA-Compliant Systematic Review.

Suboptimal reporting of randomized controlled trials on non-pharmacological therapies in Chinese medicine

Analyses of repeatedly measured continuous outcomes in randomized controlled trials needed substantial improvements

The quality of reports of randomized clinical trials on traditional Chinese medicine treatments: a systematic review of articles indexed in the China National Knowledge Infrastructure database from 2005 to 2012

Reporting time toxicity in prospective cancer clinical trials: A scoping review

[Identification process of time-related bias in pharmacoepidemiologic research based on a scoping review]

Assessing treatment effects with adjusted restricted mean time lost in observational competing risks data

Methodology and reporting characteristics of studies using interrupted time series design in healthcare

Two Stage Least Squares with Time-Varying Instruments: An Application to an Evaluation of Treatment Intensification for Type-2 Diabetes

A Cross-sectional literature survey showed the reporting quality of multicenter randomized controlled trials should be improved

Rationality of Time-Dependent Antimicrobial Use in Intensive Care Units in China: A Nationwide Cross-Sectional Survey.