Abstract:Incorporating sparsity priors in learning tasks can give rise to simple, and interpretable models for complex high dimensional data. Sparse models have found widespread use in structure discovery, recovering data from corruptions, and a variety of large scale unsupervised and supervised learning problems. Assuming the availability of sufficient data, these methods infer dictionaries for sparse representations by optimizing for high-fidelity reconstruction. In most scenarios, the reconstruction quality is measured using the squared Euclidean distance, and efficient algorithms have been developed for both batch and online learning cases. However, new application domains motivate looking beyond conventional loss functions. For example, robust loss functions such as $\ell_1$ and Huber are useful in learning outlier-resilient models, and the quantile loss is beneficial in discovering structures that are the representative of a particular quantile. These new applications motivate our work in generalizing sparse learning to a broad class of convex loss functions. In particular, we consider the class of piecewise linear quadratic (PLQ) cost functions that includes Huber, as well as $\ell_1$, quantile, Vapnik, hinge loss, and smoothed variants of these penalties. We propose an algorithm to learn dictionaries and obtain sparse codes when the data reconstruction fidelity is measured using any smooth PLQ cost function. We provide convergence guarantees for the proposed algorithm, and demonstrate the convergence behavior using empirical experiments. Furthermore, we present three case studies that require the use of PLQ cost functions: (i) robust image modeling, (ii) tag refinement for image annotation and retrieval and (iii) computing empirical confidence limits for subspace clustering.

L0Learn: A Scalable Package for Sparse Learning using L0 Regularization

Lass-0: sparse non-convex regression by local search

Sparsifying Parametric Models with L0 Regularization

A multilevel framework for sparse optimization with application to inverse covariance estimation and logistic regression

Beyond L2-Loss Functions for Learning Sparse Models

Safe Feature Elimination for the LASSO and Sparse Supervised Learning Problems

Constrained Low-Rank Learning Using Least Squares-Based Regularization

A Sparse-Group Lasso

Learning Sparse Neural Networks through L0 Regularization

Fast Learning with Nonconvex L1-2 Regularization.

DeepHoyer: Learning Sparser Neural Network with Differentiable Scale-Invariant Sparsity Measures

Lasso with Latents: Efficient Estimation, Covariate Rescaling, and Computational-Statistical Gaps

Scaled sparse linear regression

BackboneLearn: A Library for Scaling Mixed-Integer Optimization-Based Machine Learning

Efficient Feature Selection via $\ell _{2, 0}$ℓ2, 0-norm Constrained Sparse Regression.

SOL: A Library for Scalable Online Learning Algorithms

Efficient sparse Hessian based algorithms for the clustered lasso problem.

GIST: General Iterative Shrinkage and Thresholding for Non-convex Sparse Learning

L0-Regularized Learning for High-Dimensional Additive Hazards Regression

A Fast and Scalable Pathwise-Solver for Group Lasso and Elastic Net Penalized Regression via Block-Coordinate Descent

Learning for Robust Combinatorial Optimization: Algorithm and Application