Abstract:Logistic regression is one of the most popular methods in binary classification, wherein estimation of model parameters is carried out by solving the maximum likelihood (ML) optimization problem, and the ML estimator is defined to be the optimal solution of this problem. It is well known that the ML estimator exists when the data is non-separable, but fails to exist when the data is separable. First-order methods are the algorithms of choice for solving large-scale instances of the logistic regression problem. In this paper, we introduce a pair of condition numbers that measure the degree of non-separability or separability of a given dataset in the setting of binary classification, and we study how these condition numbers relate to and inform the properties and the convergence guarantees of first-order methods. When the training data is non-separable, we show that the degree of non-separability naturally enters the analysis and informs the properties and convergence guarantees of two standard first-order methods: steepest descent (for any given norm) and stochastic gradient descent. Expanding on the work of Bach, we also show how the degree of non-separability enters into the analysis of linear convergence of steepest descent (without needing strong convexity), as well as the adaptive convergence of stochastic gradient descent. When the training data is separable, first-order methods rather curiously have good empirical success, which is not well understood in theory. In the case of separable data, we demonstrate how the degree of separability enters into the analysis of $\ell_2$ steepest descent and stochastic gradient descent for delivering approximate-maximum-margin solutions with associated computational guarantees as well. This suggests that first-order methods can lead to statistically meaningful solutions in the separable case, even though the ML solution does not exist.

Phase transitions in nonparametric regressions

Phase transitions in nonparametric regressions

Optimal errors and phase transitions in high-dimensional generalized linear models

From sparse to dense functional data in high dimensions: Revisiting phase transitions from a non-asymptotic perspective

On the entropy numbers of the mixed smoothness function classes

Benign overfitting and adaptive nonparametric regression

Estimating a Function and Its Derivatives Under a Smoothness Condition

Characterizing the minimax rate of nonparametric regression under bounded convex constraints

Condition Number Analysis of Logistic Regression, and its Implications for Standard First-Order Solution Methods

Variational Multiscale Nonparametric Regression: Smooth Functions

Deterministic Inequalities for Smooth M-estimators

Asymptotic Bounds for Smoothness Parameter Estimates in Gaussian Process Interpolation

Adaptive Inference in Multivariate Nonparametric Regression Models Under Monotonicity

Learning phase transitions from regression uncertainty: A new regression-based machine learning approach for automated detection of phases of matter

Phase transition on the convergence rate of parameter estimation under an Ornstein-Uhlenbeck diffusion on a tree

Existence of solutions to the nonlinear equations characterizing the precise error of M-estimators

Learning in PINNs: Phase transition, total diffusion, and generalization

On Optimal Interpolation In Linear Regression

Higher-order non-local gradient theory of phase-transitions

Estimation and inference for minimizer and minimum of convex functions: optimality, adaptivity and uncertainty principles

Towards Optimal Problem Dependent Generalization Error Bounds in Statistical Learning Theory