Random allocation in observational data: how small but robust effects could facilitate hypothesis-free causal inference.

G. Davey Smith

DOI: https://doi.org/10.1097/EDE.0b013e31821d0426

2011-07-01

Epidemiology

Abstract:Conventional observational epidemiology has an unenviable reputation for generating false-positive findings, or “scares,” as others call them. In 1993, for example, the New York Times reported that “vitamin E greatly reduces the risk of heart disease” following simultaneous publication of 2 observational studies in the New England Journal of Medicine demonstrating that use of vitamin E supplements, even for just a few years, was associated with a substantially lower risk of coronary heart disease (CHD). Randomized controlled trials (RCTs)—testing precisely the same hypothesis—revealed no reduction in risk at all. As I write this, UK news media are reporting that “junk food makes your kids dumb,” in response to a paper reporting that children given a poor diet at age 3 had lower IQ scores 5 years later. In the latter case, however, no RCTs are ever likely to be carried out, and the status of the finding will remain liminal. The ratio of false-positive to false-negative (FP:FN) published findings in traditional epidemiology is very high, John Ioannidis and colleagues argue. But what exactly are “false-positive” findings in the context of conventional observational epidemiology, and how can their prevalence be quantified? My suspicion is that use of vitamin E supplements would—in most contexts—be associated with lower risk of CHD, because of substantial confounding by a myriad of measured and unmeasured factors related to such a socially and behaviorally patterned exposure. The same will apply to children fed on junk food and their later intelligence. The inability to substantially “control” statistically for confounding in many situations—due to measurement error in assessed confounders and omission of others—remains underappreciated. In the sense that these associations do exist, they should perhaps not be called false positives; they are false positives only if they are taken to be indicators of underlying causal effects. Noncausal but replicable observational associations could clearly be of considerable value through prediction of disease risk, and targeting of preventive measures to those who can benefit most. For the holy grail of epidemiology—identifying causes of disease—they are a disappointment, however. A wide range of approaches—from formally comparing associations in contexts where confounding structures differ, utilizing correlates of the exposure under study that are not plausible causes and natural experiments, through to formal instrumental variables methods—offer greater hope for reliable causal inference than plowing on with traditional approaches and keeping the FP:FN ratio high. Ioannidis and colleagues reiterate the low positive predictive value of a nominally “significant” P, something I (along with a generation of epidemiologists, I imagine) first encountered in Michael Oakes’ seminal “Statistical Inference, although (as has been