Abstract:Background Bias in the publication and reporting of research findings (referred to as publication and related bias here) poses a major threat in evidence synthesis and evidence-based decision-making. Although this bias has been well documented in clinical research, little is known about its occurrence and magnitude in health services and delivery research. Objectives To obtain empirical evidence on publication and related bias in quantitative health services and delivery research; to examine current practice in detecting/mitigating this bias in health services and delivery research systematic reviews; and to explore stakeholders’ perception and experiences concerning such bias. Methods The project included five distinct but interrelated work packages. Work package 1 was a systematic review of empirical and methodological studies. Work package 2 involved a survey (meta-epidemiological study) of randomly selected systematic reviews of health services and delivery research topics ( n = 200) to evaluate current practice in the assessment of publication and outcome reporting bias during evidence synthesis. Work package 3 included four case studies to explore the applicability of statistical methods for detecting such bias in health services and delivery research. In work package 4 we followed up four cohorts of health services and delivery research studies (total n = 300) to ascertain their publication status, and examined whether publication status was associated with statistical significance or perceived ‘positivity’ of study findings. Work package 5 involved key informant interviews with diverse health services and delivery research stakeholders ( n = 24), and a focus group discussion with patient and service user representatives ( n = 8). Results We identified only four studies that set out to investigate publication and related bias in health services and delivery research in work package 1. Three of these studies focused on health informatics research and one concerned health economics. All four studies reported evidence of the existence of this bias, but had methodological weaknesses. We also identified three health services and delivery research systematic reviews in which findings were compared between published and grey/unpublished literature. These reviews found that the quality and volume of evidence and effect estimates sometimes differed significantly between published and unpublished literature. Work package 2 showed low prevalence of considering/assessing publication (43%) and outcome reporting (17%) bias in health services and delivery research systematic reviews. The prevalence was lower among reviews of associations than among reviews of interventions. The case studies in work package 3 highlighted limitations in current methods for detecting these biases due to heterogeneity and potential confounders. Follow-up of health services and delivery research cohorts in work package 4 showed positive association between publication status and having statistically significant or positive findings. Diverse views concerning publication and related bias and insights into how features of health services and delivery research might influence its occurrence were uncovered through the interviews with health services and delivery research stakeholders and focus group discussion conducted in work package 5. Conclusions This study provided prima facie evidence on publication and related bias in quantitative health services and delivery research. This bias does appear to exist, but its prevalence and impact may vary depending on study characteristics, such as study design, and motivation for conducting the evaluation. Emphasis on methodological novelty and focus beyond summative assessments may mitigate/lessen the risk of such bias in health services and delivery research. Methodological and epistemological diversity in health services and delivery research and changing landscape in research publication need to be considered when interpreting the evidence. Collection of further empirical evidence and exploration of optimal health services and delivery research practice are required. Study registration This study is registered as PROSPERO CRD42016052333 and CRD42016052366. Funding This project was funded by the National Institute for Health Research (NIHR) Health Services and Delivery Research programme and will be published in full in Health Services and Delivery Research ; Vol. 8, No. 33. See the NIHR Journals Library website for further project information.

Investigating and dealing with publication bias and other reporting biases in meta‐analyses of health research: A review

Footprint of publication selection bias on meta‐analyses in medicine, environmental sciences, psychology, and economics

Testing and adjusting for publication bias

Empirical Comparison of Publication Bias Tests in Meta-Analysis

P Value–driven Methods Were Underpowered to Detect Publication Bias: Analysis of Cochrane Review Meta-Analyses

Publication bias examined in meta-analyses from psychology and medicine: A meta-meta-analysis

Publication Bias and Dissemination of Clinical Research

Statistical controversies in clinical research: publication bias evaluations are not routinely conducted in clinical oncology systematic reviews

Systematic review finds that study data not published in full text articles have unclear impact on meta-analyses results in medical research

Investigation of bias due to selective inclusion of study effect estimates in meta‐analyses of nutrition research

Publication and related bias in quantitative health services and delivery research: a multimethod study

Assessing risk of bias due to missing results in a synthesis

Investigation of publication bias in meta-analyses of diagnostic test accuracy: a meta-epidemiological study

Statistical Methods for Dealing with Publication Bias in Meta-Analysis

Sensitivity analysis for publication bias in meta‐analyses

Estimating the change in meta-analytic effect size estimates after the application of publication bias adjustment methods.

Robust Point and Variance Estimation for Meta‐analyses with Selective Reporting and Dependent Effect Sizes

Publication Bias in the Organizational Sciences

Using selection models to assess sensitivity to publication bias: A tutorial and call for more routine use

Estimating the extent of selective reporting: An application to economics

Tools for assessing risk of reporting biases in studies and syntheses of studies: a systematic review