Abstract:Gaussian covariance graph model is a popular model in revealing underlying dependency structures among random variables. A Bayesian approach to the estimation of covariance structures uses priors that force zeros on some off-diagonal entries of covariance matrices and put a positive definite constraint on matrices. In this paper, we consider a spike and slab prior on off-diagonal entries, which uses a mixture of point-mass and normal distribution. The point-mass naturally introduces sparsity to covariance structures so that the resulting posterior from this prior renders covariance structure learning. Under this prior, we calculate posterior model probabilities of covariance structures using Laplace approximation. We show that the error due to Laplace approximation becomes asymptotically marginal at some rate depending on the posterior convergence rate of covariance matrix under the Frobenius norm. With the approximated posterior model probabilities, we propose a new framework for estimating a covariance structure. Since the Laplace approximation is done around the mode of conditional posterior of covariance matrix, which cannot be obtained in the closed form, we propose a block coordinate descent algorithm to find the mode and show that the covariance matrix can be estimated using this algorithm once the structure is chosen. Through a simulation study based on five numerical models, we show that the proposed method outperforms graphical lasso and sample covariance matrix in terms of root mean squared error, max norm, spectral norm, specificity, and sensitivity. Also, the advantage of the proposed method is demonstrated in terms of accuracy compared to our competitors when it is applied to linear discriminant analysis (LDA) classification to breast cancer diagnostic dataset.

What problem does this paper attempt to address?

The problem that this paper attempts to solve is to estimate the structure of sparse covariance matrices on high - dimensional data sets. Specifically, the paper focuses on how to effectively estimate the structure of the covariance matrix while maintaining the positive definiteness of the matrix under the premise of introducing sparsity. This problem is very important in multivariate data analysis because it helps to reveal the potential dependence relationships between variables and plays a key role in various multivariate statistical inferences such as principal component analysis (PCA), linear discriminant analysis (LDA) and time - series analysis. ### Main contributions of the paper 1. **Introducing a new prior distribution**: - The paper proposes a spike - and - slab prior using a mixture of point - mass and normal distribution for the off - diagonal elements of the covariance matrix. This prior naturally introduces sparsity, making it possible to learn the covariance structure. 2. **Laplace approximation**: - In order to calculate the posterior model probability of the covariance structure, the paper adopts the Laplace approximation method. Under certain conditions, the error of this method will asymptotically become negligible, so that the approximation result has high accuracy. 3. **New framework and algorithm**: - Based on the posterior model probability obtained by the Laplace approximation, the paper proposes a new framework to estimate the covariance structure. In addition, a block coordinate descent algorithm is proposed to find the conditional posterior mode and thus estimate the covariance matrix. 4. **Performance evaluation**: - Through simulation studies based on five numerical models, the paper shows that the proposed method is superior to the graphical lasso and the sample covariance matrix in terms of root - mean - square error, maximum norm, spectral norm, specificity and sensitivity. In addition, in the linear discriminant analysis (LDA) classification task on the breast cancer diagnosis data set, the proposed method also shows higher accuracy. ### Technical details - **Prior setting**: - The paper uses point - mass prior and exponential prior, which are applied to the off - diagonal elements and diagonal elements of the covariance matrix respectively. This prior setting enables the model to naturally introduce sparsity without the need for complex adjustments like the continuous spike - and - slab prior. - **Laplace approximation**: - The Laplace approximation is a commonly used approximation method in Bayesian statistics for calculating the posterior probability of complex models. The paper proves that under this specific prior setting, the error of the Laplace approximation will asymptotically become negligible as the sample size increases. - **Block coordinate descent algorithm**: - Since the Laplace approximation is carried out near the conditional posterior mode, and this mode cannot be obtained in a closed form, the paper proposes a block coordinate descent algorithm to find this mode. The algorithm gradually approaches the optimal solution by iteratively updating each block of the covariance matrix. ### Conclusion By introducing a new prior distribution and the Laplace approximation method, this paper provides an effective framework for estimating the structure of sparse covariance matrices in high - dimensional data sets. The experimental results show that the proposed method is superior to existing methods in multiple indicators and has high practical value.

Covariance Structure Estimation with Laplace Approximation

Sparse estimation of a covariance matrix

Fast Laplace Approximation for Sparse Bayesian Spike and Slab Models

Covariance Matrix Estimation Via Network Structure

Model-based Clustering with Sparse Covariance Matrices

Posterior asymptotics of high-dimensional spiked covariance model with inverse-Wishart prior

Bayesian Nonparametric Covariance Regression

Towards a sparse, scalable, and stably positive definite (inverse) covariance estimator

Covariance Model with General Linear Structure and Divergent Parameters

Maximum a posteriori estimation in graphical models using local linear approximation

Covariance Structure Regularization Via Entropy Loss Function.

Posterior Covariance Structures in Gaussian Processes

Bayesian Sparse Covariance Structure Analysis for Correlated Count Data

Two new algorithms for maximum likelihood estimation of sparse covariance matrices with applications to graphical modeling

Structured prior distributions for the covariance matrix in latent factor models

A Best Linear Empirical Bayes Method for High-Dimensional Covariance Matrix Estimation

Blessing of dimension in Bayesian inference on covariance matrices

Bayesian inference for a covariance matrix

A convex framework for high-dimensional sparse Cholesky based covariance estimation

Approximate Bayesian inference for latent Gaussian models by using integrated nested Laplace approximations

A Constrained L1 Minimization Approach to Sparse Precision Matrix Estimation