Collapsibility of Logistic-Regression Coefficients

JH GUO,Z GENG

DOI: https://doi.org/10.1111/j.2517-6161.1995.tb02029.x

1995-01-01

Abstract:In this paper we discuss collapsibility of logistic regression coefficients over a background variable and present necessary and sufficient conditions for collapsibility.

What problem does this paper attempt to address?

Residuals and Regression Diagnostics: Focusing on Logistic Regression

Zhongheng Zhang

DOI: https://doi.org/10.21037/atm.2016.03.36

IF: 3.616

2016-01-01

Annals of Translational Medicine

Abstract:Up to now I have introduced most steps in regression model building and validation. The last step is to check whether there are observations that have significant impact on model coefficient and specification. The article firstly describes plotting Pearson residual against predictors. Such plots are helpful in identifying nonlinearity and provide hints on how to transform predictors. Next, I focus on observations of outlier, leverage and influence that may have significant impact on model building. Outlier is such an observation that its response value is unusual conditional on covariate pattern. Leverage is an observation with covariate pattern that is far away from the regressor space. Influence is the product of outlier and leverage. That is, when influential observation is dropped from the model, there will be a significant shift of the coefficient. Summary statistics for outlier, leverage and influence are studentized residuals, hat values and Cook's distance. They can be easily visualized with graphs and formally tested using the car package.
Collapsibility of relative risk in contingency tables with a response variable

Z. Geng

DOI: https://doi.org/10.1111/j.2517-6161.1992.tb01898.x

1992-01-01

Abstract:We consider the collapsibility of relative risks in contingency tables with a polytomous response variable. Simple collapsibility of relative risks means that they are consistent over all levels of a collapsed variable and also coincide with the corresponding marginal relative risk. Strong collapsibility of relative risks means that they retain simple collapsibility no matter how the variable is partially collapsed. We present necessary and sufficient conditions for simple and strong collapsibilities of relative risks.
Consecutive collapsibility of odds ratios over an ordinal background variable

Jianhua Guo,Zhi Geng,Wing-Kam Fung

DOI: https://doi.org/10.1006/jmva.2000.1957

IF: 1.387

2001-01-01

Journal of Multivariate Analysis

Abstract:G. R. Ducharme and Y. Lepage (1986, J. Roy. Statist. Soc. Ser. B 48 , 197–205) presented the strong collapsibility of odds ratio in 2×2× K tables. However, the concept is not suitable for an ordinal background variable since it is meaningless to pool nonadjacent levels in this case. In this paper, we introduce the consecutive collapsibility of odds ratios over an ordinal background in an I × J × K table and present necessary and sufficient conditions for consecutive collapsibility of odds ratios. The conditions are important for guiding the possible merging of consecutive levels of the background.
Collapsibility of distribution dependence

Zongming Ma,Xianchao Xie,Zhi Geng

DOI: https://doi.org/10.1111/j.1467-9868.2005.00536.x

2006-01-01

Abstract:Cox and Wermuth proposed that the partial derivative of the conditional distribution function of a random variable Y given another X is used for measuring association between two variables with arbitrary distributions. The paper presents a necessary and sufficient condition for uniform collapsibility of this association measure over a third variable W.
Condition Number Analysis of Logistic Regression, and its Implications for Standard First-Order Solution Methods

Robert M. Freund,Paul Grigas,Rahul Mazumder

DOI: https://doi.org/10.48550/arXiv.1810.08727

2018-10-20

Abstract:Logistic regression is one of the most popular methods in binary classification, wherein estimation of model parameters is carried out by solving the maximum likelihood (ML) optimization problem, and the ML estimator is defined to be the optimal solution of this problem. It is well known that the ML estimator exists when the data is non-separable, but fails to exist when the data is separable. First-order methods are the algorithms of choice for solving large-scale instances of the logistic regression problem. In this paper, we introduce a pair of condition numbers that measure the degree of non-separability or separability of a given dataset in the setting of binary classification, and we study how these condition numbers relate to and inform the properties and the convergence guarantees of first-order methods. When the training data is non-separable, we show that the degree of non-separability naturally enters the analysis and informs the properties and convergence guarantees of two standard first-order methods: steepest descent (for any given norm) and stochastic gradient descent. Expanding on the work of Bach, we also show how the degree of non-separability enters into the analysis of linear convergence of steepest descent (without needing strong convexity), as well as the adaptive convergence of stochastic gradient descent. When the training data is separable, first-order methods rather curiously have good empirical success, which is not well understood in theory. In the case of separable data, we demonstrate how the degree of separability enters into the analysis of $\ell_2$ steepest descent and stochastic gradient descent for delivering approximate-maximum-margin solutions with associated computational guarantees as well. This suggests that first-order methods can lead to statistically meaningful solutions in the separable case, even though the ML solution does not exist.

Optimization and Control,Machine Learning,Computation
Conditional Independence Collapsibility for Acyclic Directed Mixed Graph Models

Weihua Li,Yi Sun,Pei Heng

DOI: https://doi.org/10.1016/j.ijar.2024.109208

IF: 4.452

2024-05-01

International Journal of Approximate Reasoning

Abstract:Collapsibility refers to the property that, when marginalizing over some variables that are not of interest from the full model, the resulting marginal model of the remaining variables is equivalent to the local model induced by the subgraph on these variables. This means that when the marginal model satisfies collapsibility, statistical inference results based on the marginal model and the local model are consistent. This has significant implications for small-sample data, modeling latent variable data, and reducing the computational complexity of statistical inference. This paper focuses on studying the conditional independence collapsibility of acyclic directed mixed graph (ADMG) models. By introducing the concept of inducing paths in ADMGs and exploring its properties, the conditional independence collapsibility of ADMGs is characterized equivalently from both graph theory and statistical perspectives.

computer science, artificial intelligence
Model Collapse Demystified: The Case of Regression

Elvis Dohmatob,Yunzhen Feng,Julia Kempe

2024-02-12

Abstract:In the era of large language models like ChatGPT, the phenomenon of "model collapse" refers to the situation whereby as a model is trained recursively on data generated from previous generations of itself over time, its performance degrades until the model eventually becomes completely useless, i.e the model collapses. In this work, we study this phenomenon in the simplified setting of kernel regression and obtain results which show a clear crossover between where the model can cope with fake data, and a regime where the model's performance completely collapses. Under polynomial decaying spectral and source conditions, we obtain modified scaling laws which exhibit new crossover phenomena from fast to slow rates. We also propose a simple strategy based on adaptive regularization to mitigate model collapse. Our theoretical results are validated with experiments.

Machine Learning,Artificial Intelligence
Testing Treatment Effects in Unconfounded Studies under Model Misspecification: Logistic Regression, Discretization, and Their Combination.

M. Z. Cangul,Y. R. Chretien,R. Gutman,D. B. Rubin

DOI: https://doi.org/10.1002/sim.3633

2009-01-01

Statistics in Medicine

Abstract:Logistic regression is commonly used to test for treatment effects in observational studies. If the distribution of a continuous covariate differs between treated and control populations, logistic regression yields an invalid hypothesis test even in an uncounfounded study if the link is not logistic. This flaw is not corrected by the commonly used technique of discretizing the covariate into intervals. A valid test can be obtained by discretization followed by regression adjustment within each interval. Copyright (C) 2009 John Wiley & Sons, Ltd.
On the logic of collapsibility for causal effect measures

Vanessa Didelez,Mats Julius Stensrud

DOI: https://doi.org/10.1002/bimj.202000305

IF: 1.715

2021-02-12

Biometrical Journal

Abstract:Biometrical Journal, EarlyView.

mathematical & computational biology,statistics & probability
Logistic Regression: Tight Bounds for Stochastic and Online Optimization

Elad Hazan,Tomer Koren,Kfir Y. Levy

DOI: https://doi.org/10.48550/arXiv.1405.3843

2014-05-15

Abstract:The logistic loss function is often advocated in machine learning and statistics as a smooth and strictly convex surrogate for the 0-1 loss. In this paper we investigate the question of whether these smoothness and convexity properties make the logistic loss preferable to other widely considered options such as the hinge loss. We show that in contrast to known asymptotic bounds, as long as the number of prediction/optimization iterations is sub exponential, the logistic loss provides no improvement over a generic non-smooth loss function such as the hinge loss. In particular we show that the convergence rate of stochastic logistic optimization is bounded from below by a polynomial in the diameter of the decision set and the number of prediction iterations, and provide a matching tight upper bound. This resolves the COLT open problem of McMahan and Streeter (2012).

Machine Learning
Diagnosing Multicollinearity of Logistic Regression Model

N. A. M. R. Senaviratna,T. M. J. A. Cooray

DOI: https://doi.org/10.9734/ajpas/2019/v5i230132

2019-10-01

Asian Journal of Probability and Statistics

Abstract:One of the key problems arises in binary logistic regression model is that explanatory variables being considered for the logistic regression model are highly correlated among themselves. Multicollinearity will cause unstable estimates and inaccurate variances that affects confidence intervals and hypothesis tests. Aim of this was to discuss some diagnostic measurements to detect multicollinearity namely tolerance, Variance Inflation Factor (VIF), condition index and variance proportions. The adapted diagnostics are illustrated with data based on a study of road accidents. Secondary data used from 2014 to 2016 in this study were acquired from the Traffic Police headquarters, Colombo in Sri Lanka. The response variable is accident severity that consists of two levels particularly grievous and non-grievous. Multicolinearity is identified by correlation matrix, tolerance and VIF values and confirmed by condition index and variance proportions. The range of solutions available for logistic regression such as increasing sample size, dropping one of the correlated variables and combining variables into an index. It is safely concluded that without increasing sample size, to omit one of the correlated variables can reduce multicollinearity considerably.
A comprehensive study of coefficient signs in weighted logistic regression

Guoping Zeng

DOI: https://doi.org/10.1016/j.heliyon.2024.e35040

IF: 3.776

2024-07-22

Heliyon

Abstract:In this paper, we explore the coefficient signs in weighted logistic regression, a variation of logistic regression that includes positive weights and is commonly used for handling uneven data sets and reject inference in credit scoring. Initially, we examine simple weighted logistic regression. Assuming full rank and overlap, we demonstrate that the slope's sign matches the sign of the difference in weighted averages of the independent variable across two groups, 1 and 0. We extend this analysis to multiple weighted logistic regression by employing two vectors: one representing the slopes and the other the differences in weighted averages of the independent variables across the groups. We establish that if one vector is zero, the other must also be zero. Additionally, we prove that if the slope vector isn't zero, the angle between these vectors will be acute. Our theoretical results can serve as a preliminary step prior to feature selection, which is important in logistic regression. Our numerical analysis further illustrates how our theoretical results can be applied to the well-known German Credit Data for reject inference. Additionally, we provide a detailed explanation of feature selection in our analysis.
Global and Simultaneous Hypothesis Testing for High-Dimensional Logistic Regression Models

Rong Ma,T. Tony Cai,Hongzhe Li

DOI: https://doi.org/10.1080/01621459.2019.1699421

IF: 4.369

2020-01-21

Journal of the American Statistical Association

Abstract:High-dimensional logistic regression is widely used in analyzing data with binary outcomes. In this article, global testing and large-scale multiple testing for the regression coefficients are considered in both single- and two-regression settings. A test statistic for testing the global null hypothesis is constructed using a generalized low-dimensional projection for bias correction and its asymptotic null distribution is derived. A lower bound for the global testing is established, which shows that the proposed test is asymptotically minimax optimal over some sparsity range. For testing the individual coefficients simultaneously, multiple testing procedures are proposed and shown to control the false discovery rate and falsely discovered variables asymptotically. Simulation studies are carried out to examine the numerical performance of the proposed tests and their superiority over existing methods. The testing procedures are also illustrated by analyzing a dataset of a metabolomics study that investigates the association between fecal metabolites and pediatric Crohn's disease and the effects of treatment on such associations. <a class="ext-link" href="https://doi.org/10.1080/01621459.2019.1699421">Supplementary materials</a> for this article are available online.

statistics & probability
On the comparison of regression coefficients across multiple logistic models with binary predictors

Luca La Rocca,Monia Lupparelli,Alberto Roverato

DOI: https://doi.org/10.1007/s00184-024-00976-y

IF: 0.96

2024-10-11

Metrika

Abstract:In many applied contexts, it is of interest to identify the extent to which a given association measure changes its value as different sets of variables are included in the analysis. We consider logistic regression models where the interest is for the effect of a focal binary explanatory variable on a specific response, and a further collection of binary covariates is available. We provide a methodological framework for the joint analysis of the full set of coefficients of the focal variable computed across all the models obtained by adding or removing predictors from the set of covariates. The result is obtained by applying a specific log-hybrid linear expansion of the joint distribution of the variables that implicitly comprises all the regression coefficients of interest. In this way, we obtain a method that allows one to verify, in a flexible way, a wide range of scientific hypotheses involving the comparison of multiple logistic regression coefficients both in nested and in non-nested models. The proposed methodology is illustrated through a test bed example and an empirical application.

statistics & probability
Some association measures and their collapsibility

Xianchao Xie,Zongming Ma,Zhi Geng

IF: 1.4

2008-01-01

Statistica Sinica

Abstract:For two variables X and Y with arbitrary distributions, we consider three general association measures, the mixed derivative of interaction, the partial derivative of the conditional distribution function and the partial derivative of the conditional expectation. The sign of an association measure between X and Y may sometimes be reversed after marginalization over a third variable W. In this paper, we first compare the stringency of these measures for evaluating a positive association. Then we present the condition for avoiding the effect reversal after marginalization over W. Further we show that a modification of the condition can be used for collapsibility of the association measures over W.
The Prevalence of Neural Collapse in Neural Multivariate Regression

George Andriopoulos,Zixuan Dong,Li Guo,Zifan Zhao,Keith Ross

2024-10-30

Abstract:Recently it has been observed that neural networks exhibit Neural Collapse (NC) during the final stage of training for the classification problem. We empirically show that multivariate regression, as employed in imitation learning and other applications, exhibits Neural Regression Collapse (NRC), a new form of neural collapse: (NRC1) The last-layer feature vectors collapse to the subspace spanned by the $n$ principal components of the feature vectors, where $n$ is the dimension of the targets (for univariate regression, $n=1$); (NRC2) The last-layer feature vectors also collapse to the subspace spanned by the last-layer weight vectors; (NRC3) The Gram matrix for the weight vectors converges to a specific functional form that depends on the covariance matrix of the targets. After empirically establishing the prevalence of (NRC1)-(NRC3) for a variety of datasets and network architectures, we provide an explanation of these phenomena by modeling the regression task in the context of the Unconstrained Feature Model (UFM), in which the last layer feature vectors are treated as free variables when minimizing the loss function. We show that when the regularization parameters in the UFM model are strictly positive, then (NRC1)-(NRC3) also emerge as solutions in the UFM optimization problem. We also show that if the regularization parameters are equal to zero, then there is no collapse. To our knowledge, this is the first empirical and theoretical study of neural collapse in the context of regression. This extension is significant not only because it broadens the applicability of neural collapse to a new category of problems but also because it suggests that the phenomena of neural collapse could be a universal behavior in deep learning.

Machine Learning,Artificial Intelligence
Finite-sample performance of the maximum likelihood estimator in logistic regression

Hugo Chardon,Matthieu Lerasle,Jaouad Mourtada

2024-11-04

Abstract:Logistic regression is a classical model for describing the probabilistic dependence of binary responses to multivariate covariates. We consider the predictive performance of the maximum likelihood estimator (MLE) for logistic regression, assessed in terms of logistic risk. We consider two questions: first, that of the existence of the MLE (which occurs when the dataset is not linearly separated), and second that of its accuracy when it exists. These properties depend on both the dimension of covariates and on the signal strength. In the case of Gaussian covariates and a well-specified logistic model, we obtain sharp non-asymptotic guarantees for the existence and excess logistic risk of the MLE. We then generalize these results in two ways: first, to non-Gaussian covariates satisfying a certain two-dimensional margin condition, and second to the general case of statistical learning with a possibly misspecified logistic model. Finally, we consider the case of a Bernoulli design, where the behavior of the MLE is highly sensitive to the parameter direction.

Statistics Theory,Machine Learning
On discretely structured logistic models and their moments

Benjamin J. Walker,Helen M. Byrne

2024-10-02

Abstract:The logistic equation is ubiquitous in applied mathematics as a minimal model of saturating growth. Here, we examine a generalisation of the logistic model to fitness-structured populations, motivated by examples that range from the ageing of individuals in a species to immune cell exhaustion by cancerous tissue. Through exploration of a range of concrete examples and a general analysis of polynomial fitness functions, we derive necessary and sufficient conditions for the dependence of the kinetics on structure to result in closed, low-dimensional moment equations that are exact. Further, we showcase how coarse-grained moment information can be used to elucidate the details of structured dynamics, with immediate potential for model selection and hypothesis testing.

Populations and Evolution
An Asymptotic Result of Conditional Logistic Regression Estimator

Zhulin He,Yuyuan Ouyang

DOI: https://doi.org/10.48550/arXiv.2005.06158

2020-05-13

Abstract:In cluster-specific studies, ordinary logistic regression and conditional logistic regression for binary outcomes provide maximum likelihood estimator (MLE) and conditional maximum likelihood estimator (CMLE), respectively. In this paper, we show that CMLE is approaching to MLE asymptotically when each individual data point is replicated infinitely many times. Our theoretical derivation is based on the observation that a term appearing in the conditional average log-likelihood function is the coefficient of a polynomial, and hence can be transformed to a complex integral by Cauchy's differentiation formula. The asymptotic analysis of the complex integral can then be performed using the classical method of steepest descent. Our result implies that CMLE can be biased if individual weights are multiplied with a constant, and that we should be cautious when assigning weights to cluster-specific studies.

Statistics Theory
Uncertain logistic regression models

Jinling Gao,Zengtai Gong

DOI: https://doi.org/10.3934/math.2024512

2024-01-01

AIMS Mathematics

Abstract:Logistic regression is a generalized nonlinear regression analysis model and is often used for data mining, automatic disease diagnosis, economic prediction, and other fields. In this paper, we first aimed to introduce the concept of uncertain logistic regression based on the uncertainty theory, as well as investigating the likelihood function in the sense of uncertain measure to represent the likelihood of unknown parameters.

mathematics, applied

Collapsibility of Logistic-Regression Coefficients

Residuals and Regression Diagnostics: Focusing on Logistic Regression

Collapsibility of relative risk in contingency tables with a response variable

Consecutive collapsibility of odds ratios over an ordinal background variable

Collapsibility of distribution dependence

Condition Number Analysis of Logistic Regression, and its Implications for Standard First-Order Solution Methods

Conditional Independence Collapsibility for Acyclic Directed Mixed Graph Models

Model Collapse Demystified: The Case of Regression

Testing Treatment Effects in Unconfounded Studies under Model Misspecification: Logistic Regression, Discretization, and Their Combination.

On the logic of collapsibility for causal effect measures

Logistic Regression: Tight Bounds for Stochastic and Online Optimization

Diagnosing Multicollinearity of Logistic Regression Model

A comprehensive study of coefficient signs in weighted logistic regression

Global and Simultaneous Hypothesis Testing for High-Dimensional Logistic Regression Models

On the comparison of regression coefficients across multiple logistic models with binary predictors

Some association measures and their collapsibility

The Prevalence of Neural Collapse in Neural Multivariate Regression

Finite-sample performance of the maximum likelihood estimator in logistic regression

On discretely structured logistic models and their moments

An Asymptotic Result of Conditional Logistic Regression Estimator

Uncertain logistic regression models