Regression analysis in the context of designed experiments: Neglect not thy opportunity to test for position and parallelism

Stephen J. Powers

DOI: https://doi.org/10.1111/aab.12710

2021-06-23

Annals of Applied Biology

Abstract:Parallel lines regression analysis reveals separate lines as the best model for these data.

agriculture, multidisciplinary

What problem does this paper attempt to address?

A note on extending Scheffé's modified multiple-comparison procedure to other analysis situations

Xinyue Zhou,Joel R. Levin

DOI: https://doi.org/10.22237/jmasm/1099268100

2004-01-01

Journal of Modern Applied Statistical Methods

Abstract:This article extends Scheffé's modified (sequential) multiple-comparison procedure in one-way analysis-of-variance to other analysis situations, including interaction comparisons in factorial ANOVA designs, tests of partial regression coefficients in multiple-regression analysis, and comparisons of means in one-factor multivariate analyses of variance. Researchers who are concerned with maintaining familywise Type I error rates while increasing statistical power relative to the original (simultaneous) Scheffé-based procedures are encouraged to consider these improved multiple-comparison methods. Copyright © 2004 JMASM, Inc.
Understanding and avoiding the "weights of regression": Heterogeneous effects, misspecification, and longstanding solutions

Chad Hazlett,Tanvi Shinkre

DOI: https://doi.org/10.48550/arxiv.2403.03299

2024-03-05

Methodology

Abstract:Researchers in many fields endeavor to estimate treatment effects by regressing outcome data (Y) on a treatment (D) and observed confounders (X). Even absent unobserved confounding, the regression coefficient on the treatment reports a weighted average of strata-specific treatment effects (Angrist, 1998). Where heterogeneous treatment effects cannot be ruled out, the resulting coefficient is thus not generally equal to the average treatment effect (ATE), and is unlikely to be the quantity of direct scientific or policy interest. The difference between the coefficient and the ATE has led researchers to propose various interpretational, bounding, and diagnostic aids (Humphreys, 2009; Aronow and Samii, 2016; Sloczynski, 2022; Chattopadhyay and Zubizarreta, 2023). We note that the linear regression of Y on D and X can be misspecified when the treatment effect is heterogeneous in X. The "weights of regression", for which we provide a new (more general) expression, simply characterize how the OLS coefficient will depart from the ATE under the misspecification resulting from unmodeled treatment effect heterogeneity. Consequently, a natural alternative to suffering these weights is to address the misspecification that gives rise to them. For investigators committed to linear approaches, we propose relying on the slightly weaker assumption that the potential outcomes are linear in X. Numerous well-known estimators are unbiased for the ATE under this assumption, namely regression-imputation/g-computation/T-learner, regression with an interaction of the treatment and covariates (Lin, 2013), and balancing weights. Any of these approaches avoid the apparent weighting problem of the misspecified linear regression, at an efficiency cost that will be small when there are few covariates relative to sample size. We demonstrate these lessons using simulations in observational and experimental settings.
Residuals and Regression Diagnostics: Focusing on Logistic Regression

Zhongheng Zhang

DOI: https://doi.org/10.21037/atm.2016.03.36

IF: 3.616

2016-01-01

Annals of Translational Medicine

Abstract:Up to now I have introduced most steps in regression model building and validation. The last step is to check whether there are observations that have significant impact on model coefficient and specification. The article firstly describes plotting Pearson residual against predictors. Such plots are helpful in identifying nonlinearity and provide hints on how to transform predictors. Next, I focus on observations of outlier, leverage and influence that may have significant impact on model building. Outlier is such an observation that its response value is unusual conditional on covariate pattern. Leverage is an observation with covariate pattern that is far away from the regressor space. Influence is the product of outlier and leverage. That is, when influential observation is dropped from the model, there will be a significant shift of the coefficient. Summary statistics for outlier, leverage and influence are studentized residuals, hat values and Cook's distance. They can be easily visualized with graphs and formally tested using the car package.
An Empirical Comparison of Parametric and Permutation Tests for Regression Analysis of Randomized Experiments

Kellie Ottoboni,Fraser Lewis,Luigi Salmaso

DOI: https://doi.org/10.1080/19466315.2018.1458648

2017-10-10

Abstract:Hypothesis tests based on linear models are widely accepted by organizations that regulate clinical trials. These tests are derived using strong assumptions about the data-generating process so that the resulting inference can be based on parametric distributions. Because these methods are well understood and robust, they are sometimes applied to data that depart from assumptions, such as ordinal integer scores. Permutation tests are a nonparametric alternative that require minimal assumptions which are often guaranteed by the randomization that was conducted. We compare analysis of covariance (ANCOVA), a special case of linear regression that incorporates stratification, to several permutation tests based on linear models that control for pretreatment covariates. In simulations of randomized experiments using models which violate some of the parametric regression assumptions, the permutation tests maintain power comparable to ANCOVA. We illustrate the use of these permutation tests alongside ANCOVA using data from a clinical trial comparing the effectiveness of two treatments for gastroesophageal reflux disease. Given the considerable costs and scientific importance of clinical trials, an additional nonparametric method, such as a linear model permutation test, may serve as a robustness check on the statistical inference for the main study endpoints.

Applications
The Research about Independence of Random Errors in Quasi-Linearity Regression Model

Wei Song,Ying Jing,Zhi-Chen Tong,Fachao Li

DOI: https://doi.org/10.1109/icmlc.2011.6016809

2011-01-01

Abstract:How to construct a regression model with good operability is the core of regression analysis. In this paper, by analyzing the efficiencies of the existing regression methods, we establish quasi-linear regression model (denoted by QRM for short) which has good structure and approximation properties; further we put forward the plan to test the independence of random errors based on residual analysis, and MATLAB gives the strategies; finally, we analyze the effectiveness of these discussions by an illustrative example.
A common statistical misunderstanding in Psychology and Neuroscience: Do we need normally distributed independent or dependent variables for linear regression to work?

Christian G. Habeck,Adam M. Brickman

DOI: https://doi.org/10.1101/305946

2018-04-24

Abstract:Abstract Our ability to draw conclusions from experiments often relies on the application of inferential statistics, such as those included within the general-linear-modeling framework. The purpose of this commentary is to direct attention a common fallacy that can be observed in paper reviews and professional exchanges in the field of Psychology and Neuroscience pertaining to multiple regression analysis. The fallacy concerns the confusion of the requirements on the residuals of a linear-regression model with requirements on the dependent or independent variables themselves. A simple thought experiment, algebraic demonstration, and simulation study can demonstrate this misconception. It is our hope that consideration of the fallacy may help clarify distinctions between technical statistical issues and conceptual interpretation of research findings.
Pursuing a Prospective Perspective

Steven Kearnes

DOI: https://doi.org/10.1016/j.trechm.2020.10.012

2020-10-29

Abstract:Retrospective testing of predictive models does not consider the real-world context in which models are deployed. Prospective validation, on the other hand, enables meaningful comparisons between data generation processes by incorporating trained models and considering the subjective decisions that affect reproducibility. Prospective experiments are essential for consistent progress in modeling.

Machine Learning
A comparison of recent approaches to the analysis of repeated measurements

H. J. Keselman,James Algina,Rhonda K. Kowalchuk,Russell D. Wolfinger

DOI: https://doi.org/10.1348/000711099158964

1999-05-01

British Journal of Mathematical and Statistical Psychology

Abstract:Looney & Stanley's (1989) recommendations regarding analysis strategies for repeated measures designs containing between‐subjects grouping variables and within‐subjects repeated measures variables were re‐examined and compared to recent analysis strategies. That is, corrected degrees of freedom univariate tests, multivariate tests, mixed model tests, and tests due to Keselman, Carriere & Lix (1993) and to Algina (1994), Huynh (1978) and Lecoutre (1991) were compared for rates of Type I error in unbalanced non‐spherical repeated measures designs having varied covariance structures and no missing data on the within‐subjects variable. Heterogeneous within‐subjects and heterogeneous within‐ and between‐subjects structures were investigated along with multivariate non‐normality. Results indicated that the tests due to Keselman et al. and Algina, Huynh and Lecoutre provided effective Type I error control whereas the default mixed model approach computed with PROC MIXED (SAS Institute, 1995) generally did not. Based on power differences, we recommend that applied researchers adopt the Welch‐James type test described by Keselman et al.

psychology, experimental, mathematical,mathematics, interdisciplinary applications,statistics & probability
Regression analyses and their particularities in observational studies

Antonia Zapf,Christian Wiessner,Inke Regina König

DOI: https://doi.org/10.3238/arztebl.m2023.0278

2024-02-23

Deutsches Ärzteblatt international

Abstract:BACKGROUND: Regression analysis is a standard method in medical research. It is often not clear, however, how the individual components of regression models are to be understood and interpreted. In this article, we provide an overview of this type of analysis and discuss its special features when used in observational studies.METHODS: Based on a selective literature review, the individual components of a regression model for differently scaled outcome variables (metric: linear regression; binary: logistic regression; time to event: Cox regression; count variable: Poisson or negative binomial regression) are explained, and their interpretation is illustrated with respect to a study on multiple sclerosis. The prerequisites for the use of each of these models, their applications, and their limitations are described in detail.RESULTS: Regression analyses are used to quantify the relation between several variables and the outcome variable. In randomized clinical trials, this flexible statistical analysis method is usually lean and prespecified. In observational studies, where there is a need to control for potential confounders, researchers with knowledge of the topic in question must collaborate with experts in statistical modeling to ensure high model quality and avoid errors. Causal diagrams are an increasingly important basis for evaluation. They should be constructed in collaboration and should differentiate between confounders, mediators, and colliders.CONCLUSION: Researchers need a basic understanding of regression models so that these models will be well defined and their findings will be fully reported and correctly interpreted.

medicine, general & internal
PRACTICAL METHOD OF REGRESSION ANALYSIS APPLIED IN PREDICTION OF PARAMETERS AND EXPERIMENT DESIGN

段浩,胡宗武,杨石,黎佑铭,王云

DOI: https://doi.org/10.3321/j.issn:1001-9669.2002.01.018

2002-01-01

Abstract:A practical method of multinomial regression analysis is put forward, in which, based on changing variables into no dimensional ones and then founding experiment matrix or form, using the limited and typical data is to erect the multinomial regression equations quickly, in order to making parameter analysis and prediction of parameters' values in a definite generalized space of variables. A example of practical application in a project is given; meanwhile, the precision of the parameter prediction has been analyzed through in comparison with the results gotten from test measure. On the other hand,based on the method of regression analysis, it is advanced how to use the method to optimize an experiment design in a definite space of values of the influential factors of a system.
Multiple linear regression is a useful alternative to traditional analyses of variance

B. K. Slinker,S. A. Glantz

DOI: https://doi.org/10.1152/ajpregu.1988.255.3.r353

1988-09-01

Abstract:Physiologists often wish to compare the effects of several different treatments on a continuous variable of interest, which requires an analysis of variance. Analysis of variance, as presented in most statistics texts, generally requires that there be no missing data and often that each sample group be the same size. Unfortunately, this requirement is rarely satisfied, and investigators are confronted with the problem of how to analyze data that do not strictly fit the traditional analysis of variance paradigm. One can avoid these pitfalls by recasting the analysis of variance as a multiple linear regression problem. When there are no missing data, the results of a traditional analysis of variance and the corresponding multiple regression problem are identical; when the sample sizes are unequal or there are missing data, one can use a regression formulation to analyze data that cannot be easily handled in a traditional analysis of variance paradigm and thus overcome a practical computational limitation of traditional analysis of variance. In addition to overcoming practical limitations of traditional analysis of variance, the multiple linear regression approach is more efficient because in one run of a statistics routine, not only is the analysis of variance done but also one obtains estimates of the size of the treatment effects (as opposed to just an indication of whether such effects are present or not), and many of the pairwise multiple comparisons are done (they are equivalent to t tests for significance of the regression parameter estimates). Finally, interaction between the different treatment factors is easier to interpret than it is in traditional analysis of variance.
Lattice Designs in Standard and Simple Implicit Multi-linear Regression

Rebecca D. Wooten

DOI: https://doi.org/10.48550/arXiv.1603.07895

2016-03-25

Abstract:Statisticians generally use ordinary least squares to minimize the random error in a subject response with respect to independent explanatory variable. However, Wooten shows illustrates how ordinary least squares can be used to minimize the random error in the system without defining a subject response. Using lattice design Wooten shows that non-response analysis is a superior alternative rotation of the pyramidal relationship between random variables and parameter estimates in multi-linear regression. Non-Response Analysis for simple linear co-linearity and Rotational Analysis in Simple Linear Regression challenge the notion of fixed effects; unity is included as a random measure (variable). The illustrations using lattice designs a mean operator that generates the standard mean and the self-weighing mean, among other point estimates with random weights; and a join that illustrates variance and covariance; and develops the measures of variance referred to as internal co-variance and base variance. These concepts are used to illustrate how these measures are used to evaluate parameter estimates in standard simple linear regression and simple implicit regression (non-response and rotational). The resulting analysis of these lattice designs show standard simple linear regression limits the relationship by consider the variance in one direction as relating to the two adjacent co-variances (standard and internal) whereas non-response analysis defines the relationship in terms of the internal co-variances and the base variance.

Methodology
Understanding Regression Analysis: An Introductory Guide

Larry D. Schroeder,David L. Sjoquist,Paula E. Stephan

DOI: https://doi.org/10.4135/9781506361628

2017-01-01

Abstract:Understanding Regression Analysis: An Introductory Guide presents the fundamentals of regression analysis, from its meaning to uses, in a concise, easy-to-read, and non-technical style. It illustrates how regression coefficients are estimated, interpreted, and used in a variety of settings within the social sciences, business, law, and public policy. Packed with applied examples and using few equations, the book walks readers through elementary material using a verbal, intuitive interpretation of regression coefficients, associated statistics, and hypothesis tests. The Second Edition features updated examples and new references to modern software output.
The partial derivative framework for substantive regression effects.

Dale S. Kim,Connor J. McCabe

DOI: https://doi.org/10.1037/met0000440

IF: 10.929

2022-02-01

Psychological Methods

Abstract:egression models are ubiquitous in the psychological sciences. The standard practice in reporting and interpreting regression models are to present and interpret coefficient estimates and the associated standard errors, confidence intervals and p-values. However, coefficient estimates have limited inferential utility if the outcome is modeled nonlinearly with respect to the substantively interpreted predictors. This is problematic in common modeling strategies, such as nonlinear predictor designs and/or generalized linear models. In the former, coefficients may correspond to product, power, log, and/or exponentially transformed units. In the latter, the relationship between the predictors and outcome are modeled via a function of the outcome, rather than the outcome in its original units. In both cases, the interpretation of the coefficients alone do not provide straightforward summaries of the data, and in fact may be misleading. We address these issues by developing a framework of regression effects by integrating two critical features. First, we explicitly model substantive variables in the units that provide the desired interpretation. Second, we use partial derivatives to summarize the relations between the substantive predictors and outcome variables to account for nonlinearities arising from modeling strategies. We show how to derive estimates and standard errors for quantities of interest in the interpretive units, as well as techniques to present the relationships between variables in meaningful ways. Finally, we provide demonstrations in both simulated and real data over a wide variety of models and estimation procedures. (PsycInfo Database Record (c) 2022 APA, all rights reserved)

psychology, multidisciplinary
On the uses and abuses of regression models: a call for reform of statistical practice and teaching

John B. Carlin,Margarita Moreno-Betancur

DOI: https://doi.org/10.48550/arXiv.2309.06668

2024-09-11

Abstract:Regression methods dominate the practice of biostatistical analysis, but biostatistical training emphasises the details of regression models and methods ahead of the purposes for which such modelling might be useful. More broadly, statistics is widely understood to provide a body of techniques for "modelling data", underpinned by what we describe as the "true model myth": that the task of the statistician/data analyst is to build a model that closely approximates the true data generating process. By way of our own historical examples and a brief review of mainstream clinical research journals, we describe how this perspective has led to a range of problems in the application of regression methods, including misguided "adjustment" for covariates, misinterpretation of regression coefficients and the widespread fitting of regression models without a clear purpose. We then outline a new approach to the teaching and application of biostatistical methods, which situates them within a framework that first requires clear definition of the substantive research question at hand within one of three categories: descriptive, predictive, or causal. Within this approach, the development and application of (multivariable) regression models, as well as other advanced biostatistical methods, should proceed differently according to the type of question. Regression methods will no doubt remain central to statistical practice as they provide a powerful tool for representing variation in a response or outcome variable as a function of "input" variables, but their conceptualisation and usage should follow from the purpose at hand.

Methodology
Model-free Study of Ordinary Least Squares Linear Regression

Arun K. Kuchibhotla,Lawrence D. Brown,Andreas Buja

DOI: https://doi.org/10.48550/arXiv.1809.10538

2018-09-27

Abstract:Ordinary least squares (OLS) linear regression is one of the most basic statistical techniques for data analysis. In the main stream literature and the statistical education, the study of linear regression is typically restricted to the case where the covariates are fixed, errors are mean zero Gaussians with variance independent of the (fixed) covariates. Even though OLS has been studied under misspecification from as early as the 1960's, the implications have not yet caught up with the main stream literature and applied sciences. The present article is an attempt at a unified viewpoint that makes the various implications of misspecification stand out.

Statistics Theory
Challenges in interpreting results from ‘multiple regression’ when there is interaction between covariates

Ian Shrier,Annabelle Redelmeier,Mireille E Schnitzer,Russell J Steele

DOI: https://doi.org/10.1136/bmjebm-2019-111225

2019-08-22

Abstract:Properly interpreting research results is the foundation of evidence-based medicine. Most observational studies use multiple regression and report adjusted effects. In randomised trials, adjusted effects are often provided when there are chance baseline imbalances. The estimates for the exposure of interest (eg, treatment) from these adjusted analyses are usually interpreted as population average causal effects (PACEs); for example, what would be the difference in the mean outcome if everyone in the population was treated versus untreated? In this paper, we show this interpretation is incorrect when there is an interaction between treatment and other variables with respect to the outcome. We provide the appropriate methods to calculate the PACE from regression analyses and also introduce alternative methods that have gained popularity over the last 20 years. Finally, we explain why researchers should be cautious when excluding interaction terms based on p values.
Introduction to Implicit Regression

Rebecca D. Wooten

DOI: https://doi.org/10.48550/arXiv.1602.00158

2016-01-30

Methodology

Abstract:Statisticians usually restrict regression to model relationships that are explicitly defined dependent and independent random variables; this paper outlines the newly developed method of non-response analysis and rotational analysis for evaluating co-dependent variables without an obvious subject response. The concepts outlined challenge the notion of fixed effects; unity is included as a random measure (variable) ignoring the assumption of independence and the degree of separation is outlined which is a measure of model quality.
Residual Analysis in Simple Linear Regression: A Practical Application with the Jarque-Bera Test

Siddamsetty Upendra,

DOI: https://doi.org/10.55041/ijsrem26596

2023-05-29

Abstract:This study provides a practical guide to residual analysis in ordinary linear regression, a basic statistical technique. Explains how to calculate and interpret residuals, which play a key role in assessing model validity. The study then focuses on the Jarque-Bera test, a diagnostic tool used to assess the normality of residuals. Through a step-by-step example, we show how to calculate the skewness and kurtosis of the residuals, and then calculate the Jarque-Bera test statistic. We highlight the importance of this test in determining whether residuals obey a normal distribution, helping researchers make reliable statistical inferences. This practical guide helps readers understand the importance of residual analysis in building robust regression models. Keywords: Residual analysis, simple linear regression, model assumptions, Ordinary Least Squares, OLS estimation, Jarque-Bera test, normality, validation, practical example.
A Tutorial on Testing, Visualizing, and Probing an Interaction Involving a Multicategorical Variable in Linear Regression Analysis

Andrew F. Hayes,Amanda K. Montoya

DOI: https://doi.org/10.1080/19312458.2016.1271116

IF: 8.044

2017-01-02

Communication Methods and Measures

Abstract:Empirical communication scholars and scientists in other fields regularly use regression models to test moderation hypotheses. When the independent variable X and moderator M are dichotomous or continuous, the practice of testing a linear moderation hypothesis using regression analysis by including the product of X and M in a model of dependent variable Y is widespread. However, many research designs include multicategorical independent variables or moderators, such as in an experiment with three or more versions of a stimulus where participants are randomly assigned to one of them. Researchers are less likely to receive training about how to properly test a moderation hypothesis using regression analysis in such a situation. In this tutorial, we explain how to test, visualize, and probe interactions involving a multicategorical variable using linear regression analysis. While presenting and discussing the fundamentals—fundamentals that are not software specific—we emphasize the use of the PROCESS macro for SPSS and SAS, as it greatly simplifies the computations and potential for error that exists when doing computations by hand or using spreadsheets based on formulas in existing books on this topic. We also introduce an iterative computational implementation of the Johnson-Neyman technique for finding regions of significance of the effect of a multicategorical independent variable when the moderator is continuous.

communication

Regression analysis in the context of designed experiments: Neglect not thy opportunity to test for position and parallelism

A note on extending Scheffé's modified multiple-comparison procedure to other analysis situations

Understanding and avoiding the "weights of regression": Heterogeneous effects, misspecification, and longstanding solutions

Residuals and Regression Diagnostics: Focusing on Logistic Regression

An Empirical Comparison of Parametric and Permutation Tests for Regression Analysis of Randomized Experiments

The Research about Independence of Random Errors in Quasi-Linearity Regression Model

A common statistical misunderstanding in Psychology and Neuroscience: Do we need normally distributed independent or dependent variables for linear regression to work?

Pursuing a Prospective Perspective

A comparison of recent approaches to the analysis of repeated measurements

Regression analyses and their particularities in observational studies

PRACTICAL METHOD OF REGRESSION ANALYSIS APPLIED IN PREDICTION OF PARAMETERS AND EXPERIMENT DESIGN

Multiple linear regression is a useful alternative to traditional analyses of variance

Lattice Designs in Standard and Simple Implicit Multi-linear Regression

Understanding Regression Analysis: An Introductory Guide

The partial derivative framework for substantive regression effects.

On the uses and abuses of regression models: a call for reform of statistical practice and teaching

Model-free Study of Ordinary Least Squares Linear Regression

Challenges in interpreting results from ‘multiple regression’ when there is interaction between covariates

Introduction to Implicit Regression

Residual Analysis in Simple Linear Regression: A Practical Application with the Jarque-Bera Test

A Tutorial on Testing, Visualizing, and Probing an Interaction Involving a Multicategorical Variable in Linear Regression Analysis