Abstract:Logistic regression is a widely used statistical model to describe the relationship between a binary response variable and predictor variables in data sets. It is often used in machine learning to identify important predictor variables. This task, variable selection, typically amounts to fitting a logistic regression model regularized by a convex combination of $\ell_1$ and $\ell_{2}^{2}$ penalties. Since modern big data sets can contain hundreds of thousands to billions of predictor variables, variable selection methods depend on efficient and robust optimization algorithms to perform well. State-of-the-art algorithms for variable selection, however, were not traditionally designed to handle big data sets; they either scale poorly in size or are prone to produce unreliable numerical results. It therefore remains challenging to perform variable selection on big data sets without access to adequate and costly computational resources. In this paper, we propose a nonlinear primal-dual algorithm that addresses these shortcomings. Specifically, we propose an iterative algorithm that provably computes a solution to a logistic regression problem regularized by an elastic net penalty in $O(T(m,n)\log(1/\epsilon))$ operations, where $\epsilon \in (0,1)$ denotes the tolerance and $T(m,n)$ denotes the number of arithmetic operations required to perform matrix-vector multiplication on a data set with $m$ samples each comprising $n$ features. This result improves on the known complexity bound of $O(\min(m^2n,mn^2)\log(1/\epsilon))$ for first-order optimization methods such as the classic primal-dual hybrid gradient or forward-backward splitting methods.

Sparse inversion for derivative of log determinant

Suboptimal subspace construction for log-determinant approximation

Fast and Accurate Log-Determinant Approximations

Representing Additive Gaussian Processes by Sparse Matrices

Computing log-likelihood and its derivatives for restricted maximum likelihood methods

On Sparse variational methods and the Kullback-Leibler divergence between stochastic processes

Large-scale Log-determinant Computation through Stochastic Chebyshev Expansions

Towards Inversion-Free Sparse Bayesian Learning: A Universal Approach

Solving Log-Determinant Optimization Problems by a Newton-CG Primal Proximal Point Algorithm

Sparse Orthogonal Variational Inference for Gaussian Processes

A Proximal Point Algorithm For Log-Determinant Optimization With Group Lasso Regularization

Sparse Inverse Problems Over Measures: Equivalence of the Conditional Gradient and Exchange Methods

Scaling Gaussian Process Regression with Derivatives

Post-Processed Posteriors for Sparse Covariances and Its Application to Global Minimum Variance Portfolio

A new dual spectral projected gradient method for log-determinant semidefinite programming with hidden clustering structures

Sparse Inducing Points in Deep Gaussian Processes: Enhancing Modeling with Denoising Diffusion Variational Inference

Efficient and robust high-dimensional sparse logistic regression via nonlinear primal-dual hybrid gradient algorithms

Log-Determinant Divergences Revisited: Alpha-Beta and Gamma Log-Det Divergences

Sparse bayesian inference with regularized gaussian distributions

Second order stochastic gradient update for Cholesky factor in Gaussian variational approximation from Stein's Lemma