Abstract:The critical importance of justifying the inclusion of covariates is a facet often overlooked in data analysis. While the incorporation of covariates typically follows informal guidelines, we argue for a comprehensive exploration of underlying principles to avoid significant statistical and interpretational challenges. Our focus is on addressing three common yet problematic practices: the indiscriminate lumping of covariates, the lack of rationale for covariate inclusion, and the oversight of potential issues in result reporting. These challenges, prevalent in neuroimaging models involving covariates such as reaction time, demographics, and morphometric measures, can introduce biases, including overestimation, underestimation, masking, sign flipping, or spurious effects. Our exploration of causal inference principles underscores the pivotal role of domain knowledge in guiding covariate selection, challenging the common reliance on statistical measures. This understanding carries implications for experimental design, model-building, and result interpretation. We draw connections between these insights and reproducibility concerns, specifically addressing the selection bias resulting from the widespread practice of strict thresholding, akin to the logical pitfall associated with "double dipping." Recommendations for robust data analysis involving covariates encompass explicit research question statements, justified covariate inclusions/exclusions, centering quantitative variables for interpretability, appropriate reporting of effect estimates, and advocating a "highlight, don't hide" approach in result reporting. These suggestions are intended to enhance the robustness, transparency, and reproducibility of covariate-driven analyses, encompassing investigations involving consortium datasets such as ABCD and UK Biobank. We discuss how researchers can use a transparent depiction of the covariate relationships to enhance the ethos of open science and promote research reproducibility.

Formal Covariate Benchmarking to Bound Omitted Variable Bias

The covariate-adjusted residual estimator and its use in both randomized trials and observational settings

Bayesian variable selection in linear regression models with instrumental variables

Relaxed covariate overlap and margin-based causal effect estimation

Instrumental variable analysis in the presence of unmeasured confounding

Making sense of sensitivity: extending omitted variable bias

Through the lens of causal inference: Decisions and pitfalls of covariate selection

Methodological considerations for novel approaches to covariate-adjusted indirect treatment comparisons

Learning sources of variability from high-dimensional observational studies

Leveraging Random Assignment to Impute Missing Covariates in Causal Studies

Covariate balancing with measurement error

Learning the Covariance of Treatment Effects Across Many Weak Experiments

Efficient Covariate Balancing for the Local Average Treatment Effect

Kernel-Distance-Based Covariate Balancing

Confounder Balancing for Instrumental Variable Regression with Latent Variable

Variance-based sensitivity analysis for weighting estimators result in more informative bounds

Instrumental variables as bias amplifiers with general outcome and confounding

Accounting for Missing Covariates in Heterogeneous Treatment Estimation

Bayesian correction for covariate measurement error: a frequentist evaluation and comparison with regression calibration

Causal Inference with High-dimensional Discrete Covariates

Energy balancing of covariate distributions