Abstract:Background: In diagnostic accuracy studies, when no reference standard test is available, a group of experts, combined in an expert panel, is often used to assess the presence of the target condition using multiple relevant pieces of patient information. Based on the expert panel judgment, the accuracy of a test or model can be determined. Methodological choices in design and analysis of the expert panel procedure have been shown to vary considerably between studies as well as the quality of reporting. Objectives: To map the current landscape of expert panels used as reference standard in diagnostic accuracy or model studies. Design: PubMed was systematically searched for eligible studies published between June 1, 2012, and October 1, 2022. Data extraction was performed by one author and, in cases of doubt, checked by another author. Study characteristics, expert panel characteristics, and expert panel methodology were extracted. Eligibility criteria: Articles were included if the diagnostic accuracy of an index test or diagnostic model was assessed using an expert panel as reference standard and the study was reported in English, Dutch, or German. Results: After initial identification of 4,078 studies, 318 were included for data extraction. Expert panels were used across numerous medical domains, of which oncology was the most common (20%). The number of experts judging the presence of the target condition in each patient was 2 or less in 29%, 3 or 4 in 55%, and 5 or more in 16% of the 318 studies. Expert panel types used were an independent panel (i.e., each expert returns a judgement without conferring with other experts in the panel) in 33% of studies, a panel using a consensus method (i.e., each case was discussed by the expert panel) in 27%, a staged (i.e., each expert independently returns a judgement and discordant cases were discussed in a consensus meeting) target condition assessment approach in 11%, and a tiebreaker (i.e., each expert independently returns a judgement and discordant cases were assessed by another expert) in 8%. The exact expert panel decision approach was unclear or not reported in 21% of studies. In 5% of studies, information about remaining uncertainty in experts about the target condition presence or absence was collected for each participant. Conclusions: There is large heterogeneity in the composition of expert panels and the way that expert panels are used as reference standard in diagnostic research. Key methodological characteristics of expert panels are frequently not reported, making it difficult to replicate or reproduce results, and potentially masking biasing factors. There is a clear need for more guidance on how to perform an expert panel procedure and specific extensions of the STARD and TRIPOD reporting guidelines when using an expert panel.

Interpreting diagnostic accuracy studies based on retrospective routinely collected data

Assessment of the accuracy of diagnostic tests: the cross-sectional study

Bayesian Updating and Sequential Testing: Overcoming Inferential Limitations of Screening Tests

Diagnostic test accuracy in longitudinal study settings: Theoretical approaches with use cases from clinical practice

Diagnostic algorithm allows for a scientifically robust and reliable retrospective diagnosis using textual evidence from mid-19th century Basel, Switzerland

Comparing the Accuracy of Screening Tests with Verification of Disease Status Restricted to Test Positives

Estimation of diagnostic test accuracy without full verification: a review of latent class methods

Diagnostic test evaluation methodology: A systematic review of methods employed to evaluate diagnostic tests in the absence of gold standard – An update

Methodological Bias and Variation of Systematic Reviews on Diagnostic Test Accuracy

Protocol for conducting a systematic review on diagnostic accuracy in clinical research

Extending Hui-Walter Framework to Correlated Outcomes with Application to Diagnosis Tests of an Eye Disease among Premature Infants.

Diagnostic Discrepancies in the Emergency Department: A Retrospective Study

On repeated diagnostic testing in screening for a medical condition: How often should the diagnostic test be repeated?

Accuracy of medical billing data against the electronic health record in the measurement of colorectal cancer screening rates

Statistical inference for diagnostic test accuracy studies with multiple comparisons

Expert panel as reference standard procedure in diagnostic accuracy studies: a systematic scoping review and methodological guidance

The diagnostic certainty levels of junior clinicians: A retrospective cohort study

Small Study Effects in Diagnostic Imaging Accuracy: A Meta-Analysis

201: THE QUALITY ASSESSMENT OF STUDIES OF DIAGNOSTIC ACCURACY PUBLISHED IN THE CANCER JOURNALS

Deceptive Measures of "Success" in Early Cancer Detection

Using Patient Experience Surveys to Identify Potential Diagnostic Safety Breakdowns: A Mixed Methods Study