Abstract:We consider the problem of learning a sparse graph under the Laplacian constrained Gaussian graphical models. This problem can be formulated as a penalized maximum likelihood estimation of the Laplacian constrained precision matrix. Like in the classical graphical lasso problem, recent works made use of the $\ell_1$-norm regularization with the goal of promoting sparsity in Laplacian constrained precision matrix estimation. However, we find that the widely used $\ell_1$-norm is not effective in imposing a sparse solution in this problem. Through empirical evidence, we observe that the number of nonzero graph weights grows with the increase of the regularization parameter. From a theoretical perspective, we prove that a large regularization parameter will surprisingly lead to a complete graph, i.e., every pair of vertices is connected by an edge. To address this issue, we introduce the nonconvex sparsity penalty, and propose a new estimator by solving a sequence of weighted $\ell_1$-norm penalized sub-problems. We establish the non-asymptotic optimization performance guarantees on both optimization error and statistical error, and prove that the proposed estimator can recover the edges correctly with a high probability. To solve each sub-problem, we develop a projected gradient descent algorithm which enjoys a linear convergence rate. Finally, an extension to learn disconnected graphs is proposed by imposing additional rank constraint. We propose a numerical algorithm based on based on the alternating direction method of multipliers, and establish its theoretical sequence convergence. Numerical experiments involving synthetic and real-world data sets demonstrate the effectiveness of the proposed method.

Convergence of a Relaxed Variable Splitting Method for Learning Sparse Neural Networks via $\ell_1, \ell_0$, and transformed-$\ell_1$ Penalties

Convergence of Projected Subgradient Method with Sparse or Low-Rank Constraints

A Simple Neural Network for Sparse Optimization with $l_1$ Regularization

GIST: General Iterative Shrinkage and Thresholding for Non-convex Sparse Learning

Learning Sparse Neural Networks through L0 Regularization

Training Sparse Neural Network by Constraining Synaptic Weight on Unit Lp Sphere

spred: Solving $L_1$ Penalty with SGD

Convergence of online gradient method for feedforward neural networks with smoothing L 1/2 regularization penalty

Does the $\ell_1$-norm Learn a Sparse Graph under Laplacian Constrained Graphical Models?

Convergence Analyses on Sparse Feedforward Neural Networks Via Group Lasso Regularization.

SPARSE DEEP NEURAL NETWORKS USING <i>L</i><sub>1,</sub>-WEIGHT NORMALIZATION

An Alternating Proximal Splitting Method With Global Convergence For Nonconvex Structured Sparsity Optimization

Nonparametric regression using over-parameterized shallow ReLU neural networks

Training a neural netwok for data reduction and better generalization

From Bayesian Sparsity to Gated Recurrent Nets

Sparse deep neural networks for nonparametric estimation in high-dimensional sparse regression

Sparse-Input Neural Network using Group Concave Regularization

Learning Sparse Visual Representations with Leaky Capped Norm Regularizers

Smoothing Proximal Gradient Method for General Structured Sparse Learning

Learning soft threshold for sparse reparameterization using gradual projection operators

A global convergence theory for deep ReLU implicit networks via over-parameterization