Abstract:Incorporating sparsity priors in learning tasks can give rise to simple, and interpretable models for complex high dimensional data. Sparse models have found widespread use in structure discovery, recovering data from corruptions, and a variety of large scale unsupervised and supervised learning problems. Assuming the availability of sufficient data, these methods infer dictionaries for sparse representations by optimizing for high-fidelity reconstruction. In most scenarios, the reconstruction quality is measured using the squared Euclidean distance, and efficient algorithms have been developed for both batch and online learning cases. However, new application domains motivate looking beyond conventional loss functions. For example, robust loss functions such as $\ell_1$ and Huber are useful in learning outlier-resilient models, and the quantile loss is beneficial in discovering structures that are the representative of a particular quantile. These new applications motivate our work in generalizing sparse learning to a broad class of convex loss functions. In particular, we consider the class of piecewise linear quadratic (PLQ) cost functions that includes Huber, as well as $\ell_1$, quantile, Vapnik, hinge loss, and smoothed variants of these penalties. We propose an algorithm to learn dictionaries and obtain sparse codes when the data reconstruction fidelity is measured using any smooth PLQ cost function. We provide convergence guarantees for the proposed algorithm, and demonstrate the convergence behavior using empirical experiments. Furthermore, we present three case studies that require the use of PLQ cost functions: (i) robust image modeling, (ii) tag refinement for image annotation and retrieval and (iii) computing empirical confidence limits for subspace clustering.

Sparse learning of maximum likelihood model for optimization of complex loss function

A Sparse MLE Approach for Joint Interference Mitigation and Data Recovery

A novel multivariate performance optimization method based on sparse coding and hyper-predictor learning

Beyond L2-Loss Functions for Learning Sparse Models

Learning Surrogate Losses

Learning Effective Loss Functions Efficiently

A Two-Stage Approach for Learning A Sparse Model with Sharp Excess Risk Analysis

Sparse Learning for Large-Scale and High-Dimensional Data: A Randomized Convex-Concave Optimization Approach

Maximum Optimality Margin: A Unified Approach for Contextual Linear Programming and Inverse Linear Programming

Minimum Error Entropy Algorithms With Sparsity Penalty Constraints

Linear Programming-Based Sparse Kernel Regression with L1-Norm Minimization for Nonlinear System Modeling

Improving Sparsity and Scalability in Regularized Nonconvex Truncated-Loss Learning Problems

Learnable Model-Driven Performance Prediction and Optimization for Imperfect MIMO System: Framework and Application

Minimax sparse logistic regression for very high-dimensional feature selection.

Sparse Signal Recovery Via Minimax-Concave Penalty and ℓ 1 -Norm Loss Function.

Feature Screening Strategy for Non-Convex Sparse Logistic Regression with Log Sum Penalty

Fast Sparse Recovery Via Non-Convex Optimization

Structured Sparsity Optimization with Non-Convex Surrogates of L2,0-Norm: A Unified Algorithmic Framework

Sparse Learning for Stochastic Composite Optimization

Impulsive Noise Recovery and Elimination: A Sparse Machine Learning Based Approach