Abstract:Logistic regression is one of the most popular methods in binary classification, wherein estimation of model parameters is carried out by solving the maximum likelihood (ML) optimization problem, and the ML estimator is defined to be the optimal solution of this problem. It is well known that the ML estimator exists when the data is non-separable, but fails to exist when the data is separable. First-order methods are the algorithms of choice for solving large-scale instances of the logistic regression problem. In this paper, we introduce a pair of condition numbers that measure the degree of non-separability or separability of a given dataset in the setting of binary classification, and we study how these condition numbers relate to and inform the properties and the convergence guarantees of first-order methods. When the training data is non-separable, we show that the degree of non-separability naturally enters the analysis and informs the properties and convergence guarantees of two standard first-order methods: steepest descent (for any given norm) and stochastic gradient descent. Expanding on the work of Bach, we also show how the degree of non-separability enters into the analysis of linear convergence of steepest descent (without needing strong convexity), as well as the adaptive convergence of stochastic gradient descent. When the training data is separable, first-order methods rather curiously have good empirical success, which is not well understood in theory. In the case of separable data, we demonstrate how the degree of separability enters into the analysis of $\ell_2$ steepest descent and stochastic gradient descent for delivering approximate-maximum-margin solutions with associated computational guarantees as well. This suggests that first-order methods can lead to statistically meaningful solutions in the separable case, even though the ML solution does not exist.

A modern maximum-likelihood theory for high-dimensional logistic regression

Maximum Likelihood Estimators in a Two Step Model for PLS

On the correspondence of deviances and maximum likelihood and interval estimates from log-linear to logistic regression modelling

Finite-sample performance of the maximum likelihood estimator in logistic regression

Global and Simultaneous Hypothesis Testing for High-Dimensional Logistic Regression Models

On the maximum likelihood estimation in general log-linear models

High-dimensional logistic regression with missing data: Imputation, regularization, and universality

Condition Number Analysis of Logistic Regression, and its Implications for Standard First-Order Solution Methods

The Impact of Regularization on High-dimensional Logistic Regression

On the existence of the maximum likelihood estimate and convergence rate under gradient descent for multi-class logistic regression

High Dimensional Logistic Regression Under Network Dependence

Multinomial Logistic Regression: Asymptotic Normality on Null Covariates in High-Dimensions

An Asymptotic Result of Conditional Logistic Regression Estimator

Logistic and multinomial-logit models: A brief review on their modifications and extensions

Nonparametric logistic regression with deep learning

Statistical Inference for High-Dimensional Generalized Linear Models With Binary Outcomes

Logistic Regression with Misclassification in Binary Outcome Variables: a Method and Software

On high-dimensional classification by sparse generalized Bayesian logistic regression

High-dimensional classification by sparse logistic regression

Essential formulae for restricted maximum likelihood and its derivatives associated with the linear mixed models

Jeffreys-prior penalty for high-dimensional logistic regression: A conjecture about aggregate bias