Abstract:Clinicians often suspect that a treatment effect can vary across individuals. However, they usually lack "evidence-based" guidance regarding potential heterogeneity of treatment effects (HTE). Potentially actionable HTE is rarely discovered in clinical trials and is widely believed (or rationalized) by researchers to be rare. Conventional statistical methods to test for possible HTE are extremely conservative and tend to reinforce this belief. In truth, though, there is no realistic way to know whether a common, or average, effect estimated from a clinical trial is relevant for all, or even most, patients. This absence of evidence, misinterpreted as evidence of absence, may be resulting in sub-optimal treatment for many individuals. We first summarize the historical context in which current statistical methods for randomized controlled trials (RCTs) were developed, focusing on the conceptual and technical limitations that shaped, and restricted, these methods. In particular, we explain how the common-effect assumption came to be virtually unchallenged. Second, we propose a simple graphical method for exploratory data analysis that can provide useful visual evidence of possible HTE. The basic approach is to display the complete distribution of outcome data rather than relying uncritically on simple summary statistics. Modern graphical methods, unavailable when statistical methods were initially formulated a century ago, now render fine-grained interrogation of the data feasible. We propose comparing observed treatment-group data to "pseudo data" engineered to mimic that which would be expected under a particular HTE model, such as the common-effect model. A clear discrepancy between the distributions of the common-effect pseudo data and the actual treatment-effect data provides prima facie evidence of HTE to motivate additional confirmatory investigation. Artificial data are used to illustrate implications of ignoring heterogeneity in practice and how the graphical method can be useful.

Behavioural science is unlikely to change the world without a heterogeneity revolution

Systematic heterogenization revisited: Increasing variation in animal experiments to improve reproducibility?

Systematic heterogenization for better reproducibility in animal experimentation

When Is Heterogeneity Actionable for Personalization?

Large Scale Replication Projects in Contemporary Psychological Research

Lifting the veil off treatment effect heterogeneity

Why meta-analyses of growth mindset and other interventions should follow best practices for examining heterogeneity: Commentary on Macnamara and Burgoyne (2023) and Burnette et al. (2023).

Heterogeneous treatment effects analysis for social scientists: A review

Avoiding the misuse of BLUP in behavioural ecology

Hits and misses in the last decade of open science: Researchers from different subfields and career stages offer personal reflections and suggestions

Methodological concerns underlying a lack of evidence for cultural heterogeneity in the replication of psychological effects

Interdisciplinary lessons and recommendations for the evaluation of replicability in behavioral sciences

Genetic heterogeneity induces nonadditive behavioural changes in Drosophila

Making more effective use of human behavioural science in conservation interventions

A Unified Survey of Treatment Effect Heterogeneity Modelling and Uplift Modelling

Exploring open science practices in behavioural public policy research

Taking the Big Leap | understanding, accessing and improving behavioural science interventions

Big-team science does not guarantee generalizability

The construct-behavior gap in behavioral decision research: A challenge beyond replicability

The Myth of the Need for Diversity Among Subjects in Theory-Testing Research: Comments on "Racial Inequality in Psychological Research" by Roberts et al. (2020)

Effect or Treatment Heterogeneity? Policy Evaluation with Aggregated and Disaggregated Treatments