Abstract:Background Bias in the publication and reporting of research findings (referred to as publication and related bias here) poses a major threat in evidence synthesis and evidence-based decision-making. Although this bias has been well documented in clinical research, little is known about its occurrence and magnitude in health services and delivery research. Objectives To obtain empirical evidence on publication and related bias in quantitative health services and delivery research; to examine current practice in detecting/mitigating this bias in health services and delivery research systematic reviews; and to explore stakeholders’ perception and experiences concerning such bias. Methods The project included five distinct but interrelated work packages. Work package 1 was a systematic review of empirical and methodological studies. Work package 2 involved a survey (meta-epidemiological study) of randomly selected systematic reviews of health services and delivery research topics ( n = 200) to evaluate current practice in the assessment of publication and outcome reporting bias during evidence synthesis. Work package 3 included four case studies to explore the applicability of statistical methods for detecting such bias in health services and delivery research. In work package 4 we followed up four cohorts of health services and delivery research studies (total n = 300) to ascertain their publication status, and examined whether publication status was associated with statistical significance or perceived ‘positivity’ of study findings. Work package 5 involved key informant interviews with diverse health services and delivery research stakeholders ( n = 24), and a focus group discussion with patient and service user representatives ( n = 8). Results We identified only four studies that set out to investigate publication and related bias in health services and delivery research in work package 1. Three of these studies focused on health informatics research and one concerned health economics. All four studies reported evidence of the existence of this bias, but had methodological weaknesses. We also identified three health services and delivery research systematic reviews in which findings were compared between published and grey/unpublished literature. These reviews found that the quality and volume of evidence and effect estimates sometimes differed significantly between published and unpublished literature. Work package 2 showed low prevalence of considering/assessing publication (43%) and outcome reporting (17%) bias in health services and delivery research systematic reviews. The prevalence was lower among reviews of associations than among reviews of interventions. The case studies in work package 3 highlighted limitations in current methods for detecting these biases due to heterogeneity and potential confounders. Follow-up of health services and delivery research cohorts in work package 4 showed positive association between publication status and having statistically significant or positive findings. Diverse views concerning publication and related bias and insights into how features of health services and delivery research might influence its occurrence were uncovered through the interviews with health services and delivery research stakeholders and focus group discussion conducted in work package 5. Conclusions This study provided prima facie evidence on publication and related bias in quantitative health services and delivery research. This bias does appear to exist, but its prevalence and impact may vary depending on study characteristics, such as study design, and motivation for conducting the evaluation. Emphasis on methodological novelty and focus beyond summative assessments may mitigate/lessen the risk of such bias in health services and delivery research. Methodological and epistemological diversity in health services and delivery research and changing landscape in research publication need to be considered when interpreting the evidence. Collection of further empirical evidence and exploration of optimal health services and delivery research practice are required. Study registration This study is registered as PROSPERO CRD42016052333 and CRD42016052366. Funding This project was funded by the National Institute for Health Research (NIHR) Health Services and Delivery Research programme and will be published in full in Health Services and Delivery Research ; Vol. 8, No. 33. See the NIHR Journals Library website for further project information.

Statistical controversies in clinical research: publication bias evaluations are not routinely conducted in clinical oncology systematic reviews

Systematic Analysis of Publication Bias in Neurosurgery Meta-Analyses

Publication Bias and Dissemination of Clinical Research

Systematic review finds that study data not published in full text articles have unclear impact on meta-analyses results in medical research

Selective Outcome Reporting in Cancer Studies: A Scoping Review

Publication and related bias in quantitative health services and delivery research: a multimethod study

Distinguishing Clinical from Statistical Significances in Contemporary Comparative Effectiveness Research

Investigation of publication bias in meta-analyses of diagnostic test accuracy: a meta-epidemiological study

Systematic bias between blinded independent central review and local assessment: literature review and analyses of 76 phase III randomised controlled trials in 45 688 patients with advanced solid tumour

Systematic reviews and meta-analyses of preclinical studies: Publication bias in laboratory animal experiments

Empirical Comparison of Publication Bias Tests in Meta-Analysis

Systematic review: Outcome reporting bias is a problem in high impact factor neurology journals

Epidemiology and Reporting Characteristics of Systematic Reviews of Biomedical Research: A Cross-Sectional Study

An Analysis of Contemporary Oncology Randomized Clinical Trials From Low/Middle-Income vs High-Income Countries

Publication bias examined in meta-analyses from psychology and medicine: A meta-meta-analysis

Publication bias in otorhinolaryngology meta-analyses in 2021

Systematic Evaluation and Comparison of Statistical Tests for Publication Bias

Design characteristics, risk of bias, and reporting of randomised controlled trials supporting approvals of cancer drugs by European Medicines Agency, 2014-16: cross sectional analysis

Quality of reporting of modern randomized controlled trials in medical oncology: a systematic review

Reporting and interpretation of results from clinical trials that did not claim a treatment difference: survey of four general medical journals

Disparity between statistical and clinical significance in published randomised controlled trials indexed in PubMed: a protocol for a cross-sectional methodological survey