Abstract:Sparse linear (or generalized linear) models combine a standard likelihood function with a sparse prior on the unknown coefficients. These priors can conveniently be expressed as a maximization over zero-mean Gaussians with different variance hyperparameters. Standard MAP estimation (Type I) involves maximizing over both the hyperparameters and coefficients, while an empirical Bayesian alternative (Type II) first marginalizes the coefficients and then maximizes over the hyperparameters, leading to a tractable posterior approximation. The underlying cost functions can be related via a dual-space framework from Wipf et al. (2011), which allows both the Type I or Type II objectives to be expressed in either coefficient or hyperparmeter space. This perspective is useful because some analyses or extensions are more conducive to development in one space or the other. Herein we consider the estimation of a trade-off parameter balancing sparsity and data fit. As this parameter is effectively a variance, natural estimators exist by assessing the problem in hyperparameter (variance) space, transitioning natural ideas from Type II to solve what is much less intuitive for Type I. In contrast, for analyses of update rules and sparsity properties of local and global solutions, as well as extensions to more general likelihood models, we can leverage coefficient-space techniques developed for Type I and apply them to Type II. For example, this allows us to prove that Type II-inspired techniques can be successful recovering sparse coefficients when unfavorable restricted isometry properties (RIP) lead to failure of popular L1 reconstructions. It also facilitates the analysis of Type II when non-Gaussian likelihood models lead to intractable integrations.

On Primal and Dual Sparsity of Markov Networks.

Primal Sparse Max-margin Markov Networks

MaxQ: Multi-Axis Query for N:M Sparsity Network

Laplace Maximum Margin Markov Networks.

Learning Best Combination for Efficient N:M Sparsity

Partially Observed Maximum Entropy Discrimination Markov Networks

Efficient Probabilistic Latent Semantic Analysis with Sparsity Control

Nonsparse learning with latent variables

Effective Sparsification of Neural Networks with Global Sparsity Constraint

Efficient Marginalization of Discrete and Structured Latent Variables via Sparsity

Minimum Variance Unbiased N:M Sparsity for the Neural Gradients

Learning Sparse Features for Classification by Mixture Models

A Discrete-Time Neurodynamic Approach to Sparsity-Constrained Nonnegative Matrix Factorization

Network Topology Inference with Sparsity and Laplacian Constraints

LLaMA-MoE v2: Exploring Sparsity of LLaMA from Perspective of Mixture-of-Experts with Post-Training

Marginal Pseudo-Likelihood Learning of Markov Network structures

Shaving Weights with Occam's Razor: Bayesian Sparsification for Neural Networks Using the Marginal Likelihood

Understanding the Landscape of Sparse Networks: A Brief Investigation

MultiMax: Sparse and Multi-Modal Attention Learning

From Bayesian Sparsity to Gated Recurrent Nets

Dual-Space Analysis of the Sparse Linear Model