Abstract:Logistic regression is one of the most popular methods in binary classification, wherein estimation of model parameters is carried out by solving the maximum likelihood (ML) optimization problem, and the ML estimator is defined to be the optimal solution of this problem. It is well known that the ML estimator exists when the data is non-separable, but fails to exist when the data is separable. First-order methods are the algorithms of choice for solving large-scale instances of the logistic regression problem. In this paper, we introduce a pair of condition numbers that measure the degree of non-separability or separability of a given dataset in the setting of binary classification, and we study how these condition numbers relate to and inform the properties and the convergence guarantees of first-order methods. When the training data is non-separable, we show that the degree of non-separability naturally enters the analysis and informs the properties and convergence guarantees of two standard first-order methods: steepest descent (for any given norm) and stochastic gradient descent. Expanding on the work of Bach, we also show how the degree of non-separability enters into the analysis of linear convergence of steepest descent (without needing strong convexity), as well as the adaptive convergence of stochastic gradient descent. When the training data is separable, first-order methods rather curiously have good empirical success, which is not well understood in theory. In the case of separable data, we demonstrate how the degree of separability enters into the analysis of $\ell_2$ steepest descent and stochastic gradient descent for delivering approximate-maximum-margin solutions with associated computational guarantees as well. This suggests that first-order methods can lead to statistically meaningful solutions in the separable case, even though the ML solution does not exist.

Condition numbers of the generalized ridge regression and its statistical estimation

Small-sample Properties for Generalized Ridge Estimator of Regression Coefficients in Linear Regression Model

Equality between two general ridge estimators and equivalence of their residual sums of squares

A new ridge type estimator and its performance for the linear regression model: Simulation and application

Condition Numbers for the K-weighted Pseudoinverse and Their Statistical Estimation

On the partial condition numbers for the indefinite least squares problem

Optimal Rates of Kernel Ridge Regression under Source Condition in Large Dimensions

Estimation of ill-conditioned models using penalized sums of squares of the residuals

Fast stochastic second-order method logarithmic in condition number

New ridge parameter estimators for the quasi-Poisson ridge regression model

Dimension free ridge regression

COMPONENTWISE CONDITION NUMBERS FOR GENERALIZED MATRIX INVERSION AND LINEAR LEAST SQUARES

How Many Machines Can We Use in Parallel Computing for Kernel Ridge Regression?

Condition number based complexity estimate for computing local extrema

Ridge estimation for uncertain autoregressive moving average model with imprecise observations

Mixed, Componentwise Condition Numbers and Small Sample Statistical Condition Estimation for Generalized Spectral Projections and Matrix Sign Functions

Characterizing Overfitting in Kernel Ridgeless Regression Through the Eigenspectrum

APPLICATION OF L-CURVE METHOD TO EQUIVALENT WEIGHT ROBUST RIDGE ESTIMAION MODEL

Efficient Algorithms for Solving Condition Number-Constrained Matrix Minimization Problems

Condition Number Analysis of Logistic Regression, and its Implications for Standard First-Order Solution Methods

EM algorithm for generalized Ridge regression with spatial covariates