Abstract:Logistic regression is commonly used for modeling dichotomous outcomes. In the classical setting, where the number of observations is much larger than the number of parameters, properties of the maximum likelihood estimator in logistic regression are well understood. Recently, Sur and Candes have studied logistic regression in the high-dimensional regime, where the number of observations and parameters are comparable, and show, among other things, that the maximum likelihood estimator is biased. In the high-dimensional regime the underlying parameter vector is often structured (sparse, block-sparse, finite-alphabet, etc.) and so in this paper we study regularized logistic regression (RLR), where a convex regularizer that encourages the desired structure is added to the negative of the log-likelihood function. An advantage of RLR is that it allows parameter recovery even for instances where the (unconstrained) maximum likelihood estimate does not exist. We provide a precise analysis of the performance of RLR via the solution of a system of six nonlinear equations, through which any performance metric of interest (mean, mean-squared error, probability of support recovery, etc.) can be explicitly computed. Our results generalize those of Sur and Candes and we provide a detailed study for the cases of $\ell_2^2$-RLR and sparse ($\ell_1$-regularized) logistic regression. In both cases, we obtain explicit expressions for various performance metrics and can find the values of the regularizer parameter that optimizes the desired performance. The theory is validated by extensive numerical simulations across a range of parameter values and problem instances.

A Naïve Bayes Regularized Logistic Regression Estimator for Low-dimensional Classification

Evolutionary Lazy Learning for Naive Bayes Classification

Sparse Bayesian Approach to Fast Learning Network for Multiclassification.

The Impact of Regularization on High-dimensional Logistic Regression

On Regularized Sparse Logistic Regression

High-dimensional classification by sparse logistic regression

ELASTIC-NET REGULARIZED HIGH-DIMENSIONAL NEGATIVE BINOMIAL REGRESSION: CONSISTENCY AND WEAK SIGNAL DETECTION

Regularized Linear Regression for Binary Classification

Local logistic regression for dimension reduction in classification

Low-rank Discriminative Regression Learning for Image Classification

Regularized Linear Discriminant Analysis Using a Nonlinear Covariance Matrix Estimator

Fractional Naive Bayes (FNB): non-convex optimization for a parsimonious weighted selective naive Bayes classifier

On low-rank regularized least squares for scalable nonlinear classification

Elastic-net Regularized High-dimensional Negative Binomial Regression: Consistency and Weak Signals Detection

Nonparametric Bayes Classification via Learning of Affine Subspaces

On high-dimensional classification by sparse generalized Bayesian logistic regression

Naive imputation implicitly regularizes high-dimensional linear models

Density-based logistic regression

A new regularization path for logistic regression via linearized Bregman

Efficient heuristics for learning scalable Bayesian network classifier from labeled and unlabeled data