Marginal Likelihood Calculation for the Gelfand–Dey and Chib Methods

Chun Liu,Qing Liu

DOI: https://doi.org/10.1016/j.econlet.2011.12.034

IF: 1.469

2011-01-01

Economics Letters

Abstract:A trade-off exists between the Gelfand and Dey (1994) and Chib (1995) methods to calculate the marginal likelihood in Bayesian estimation. Using the Markov Chain Monte Carlo method, we demonstrate that the performance of the two methods is fairly close.

What problem does this paper attempt to address?

Monte Carlo Approximation of Bayes Factors via Mixing with Surrogate Distributions

Chenguang Dai,Jun S. Liu

DOI: https://doi.org/10.48550/arXiv.1909.05922

2020-12-16

Abstract:By mixing the target posterior distribution with a surrogate distribution, of which the normalizing constant is tractable, we propose a method for estimating the marginal likelihood using the Wang-Landau algorithm. We show that a faster convergence of the proposed method can be achieved via the momentum acceleration. Two implementation strategies are detailed: (i) facilitating global jumps between the posterior and surrogate distributions via the Multiple-try Metropolis; (ii) constructing the surrogate via the variational approximation. When a surrogate is difficult to come by, we describe a new jumping mechanism for general reversible jump Markov chain Monte Carlo algorithms, which combines the Multiple-try Metropolis and a directional sampling algorithm. We illustrate the proposed methods on several statistical models, including the Log-Gaussian Cox process, the Bayesian Lasso, the logistic regression, and the g-prior Bayesian variable selection.

Computation,Methodology
Marginal Likelihood Computation for Model Selection and Hypothesis Testing: An Extensive Review

F. Llorente,L. Martino,D. Delgado,J. López-Santiago

DOI: https://doi.org/10.1137/20M1310849

IF: 10.2

2023-02-10

SIAM Review

Abstract:SIAM Review, Volume 65, Issue 1, Page 3-58, February 2023. This is an up-to-date introduction to, and overview of, marginal likelihood computation for model selection and hypothesis testing. Computing normalizing constants of probability models (or ratios of constants) is a fundamental issue in many applications in statistics, applied mathematics, signal processing, and machine learning. This article provides a comprehensive study of the state of the art of the topic. We highlight limitations, benefits, connections, and differences among the different techniques. Problems and possible solutions with the use of improper priors are also described. Some of the most relevant methodologies are compared through theoretical comparisons and numerical experiments.

mathematics, applied
Poisson and Gamma Model Marginalisation and Marginal Likelihood calculation using Moment-generating Functions

Siyang Li,David van Dyk,Maximilian Autenrieth

2024-09-17

Abstract:We present a new analytical method to derive the likelihood function that has the population of parameters marginalised out in Bayesian hierarchical models. This method is also useful to find the marginal likelihoods in Bayesian models or in random-effect linear mixed models. The key to this method is to take high-order (sometimes fractional) derivatives of the prior moment-generating function if particular existence and differentiability conditions hold. In particular, this analytical method assumes that the likelihood is either Poisson or gamma. Under Poisson likelihoods, the observed Poisson count determines the order of the derivative. Under gamma likelihoods, the shape parameter, which is assumed to be known, determines the order of the fractional derivative. We also present some examples validating this new analytical method.

Methodology,Statistics Theory
Improved marginal likelihood estimation via power posteriors and importance sampling

Yong Li,Nianling Wang,Jun Yu

DOI: https://doi.org/10.1016/j.jeconom.2021.11.009

IF: 3.363

2021-12-01

Journal of Econometrics

Abstract:Power posteriors have become popular in estimating the marginal likelihood of a Bayesian model. A power posterior is referred to as the posterior distribution that is proportional to the likelihood raised to a power b ∈ [ 0 , 1 ] . Important power-posterior-based algorithms include thermodynamic integration (TI) of Friel and Pettitt (2008) and steppingstone sampling (SS) of Xie et al. (2011). In this paper, it is shown that the Bernstein–von Mises (BvM) theorem holds for power posteriors under regularity conditions. Due to the BvM theorem, power posteriors, when adjusted by the square root of the auxiliary constant, have the same limit distribution as the original posterior distribution, facilitating the implementation of the modified TI and SS methods via importance sampling. Unlike the TI and SS methods that require repeated sampling from the power posteriors, the modified methods only need the original posterior output and hence, are computationally more efficient. Moreover, they completely avoid the coding efforts associated with sampling from the power posteriors. Primitive conditions, under which the TI and modified TI algorithms can produce consistent estimators of the marginal likelihood, are provided. The numerical efficiency of the proposed methods is illustrated using two models.

economics,social sciences, mathematical methods,mathematics, interdisciplinary applications
Identifying Structural Breaks by Marginal Likelihood

Liu Chun,Jin Hongfei,Pan Huifeng

DOI: https://doi.org/10.3969/j.issn.1002-4565.2010.11.015

2010-01-01

Abstract:How to identify structural breaks in the financial market is a hot but difficult task.In the Bayesian framework,we use marginal likelihood to identify the structural breaks in the volatility model.Using Monte Carol method,we find our method works well.Furthermore,the performance of the Gelfand-Dey method is very close to the Chib method to calculate the marginal likelihood.Applied to daily S&P 500 data,we find strong evidence of a structural break in the US stock Market.
Lindeberg's Central Limit Theorems for Martingale-Like Sequences under Sub-Linear Expectations

Zhang Li-Xin

DOI: https://doi.org/10.1007/s11425-018-9556-7

2020-01-01

Science China Mathematics

Abstract:The central limit theorem of martingales is the fundamental tool for studying the convergence of stochastic processes, especially stochastic integrals and differential equations. In this paper, the central limit theorem and the functional central limit theorem are obtained for martingale-like random variables under the sub-linear expectation. As applications, the Lindeberg’s central limit theorem is obtained for independent but not necessarily identically distributed random variables, and a new proof of the Lévy characterization of a GBrownian motion without using stochastic calculus is given. For proving the results, Rosenthal’s inequality and the exponential inequality for the martingale-like random variables are established.
Non-Gaussian Quasi Maximum Likelihood Estimation of GARCH Models

Lei Qi,Dacheng Xiu,Jianqing Fan

DOI: https://doi.org/10.48550/arXiv.1001.3895

2010-06-14

Abstract:The non-Gaussian quasi maximum likelihood estimator is frequently used in GARCH models with intension to improve the efficiency of the GARCH parameters. However, unless the quasi-likelihood happens to be the true one, non-Gaussian QMLE methods suffers inconsistency even if shape parameters in the quasi-likelihood are estimated. To correct this bias, we identify an unknown scale parameter that is critical to the consistent estimation of non-Gaussian QMLE, and propose a two-step non-Gaussian QMLE (2SNG-QMLE) for estimation of the scale parameter and GARCH parameters. This novel approach is consistent and asymptotically normal. Moreover, it has higher efficiency than the Gaussian QMLE, particularly when the innovation error has heavy tails. Two extensions are proposed to further improve the efficiency of 2SNG-QMLE. The impact of relative heaviness of tails of the innovation and quasi-likelihood distributions on the asymptotic efficiency has been thoroughly investigated. Monte Carlo simulations and an empirical study confirm the advantages of the proposed approach.

Methodology,Statistics Theory,Applications
Variance Reduction for Generalized Likelihood Ratio Method by Conditional Monte Carlo and Randomized Quasi-Monte Carlo Methods

Yijie Peng,Michael C. Fu,Jiaqiao Hu,Pierre L'Ecuyer,Bruno Tuffin

DOI: https://doi.org/10.1016/j.jmse.2022.02.002

2022-01-01

Journal of Management Science and Engineering

Abstract:The generalized likelihood ratio (GLR) method is a recently introduced gradient estimation method for handling discontinuities in a wide range of sample performances. We put the GLR methods from previous work into a single framework, simplify regularity conditions to justify the unbiasedness of GLR, and relax some of those conditions that are difficult to verify in practice. Moreover, we combine GLR with conditional Monte Carlo methods and randomized quasi-Monte Carlo methods to reduce the variance. Numerical experiments show that variance reduction could be significant in various applications.
Approximate Marginal Likelihood Inference in Mixed Models for Grouped Data

Alex Stringer

DOI: https://doi.org/10.48550/arXiv.2310.01589

2024-11-12

Abstract:A method is introduced for approximate marginal likelihood inference via adaptive Gaussian quadrature in mixed models with a single grouping factor. The core technical contribution is an algorithm for computing the exact gradient of the approximate log marginal likelihood. This leads to efficient maximum likelihood via quasi-Newton optimization that is demonstrated to be faster than existing approaches based on finite-differenced gradients or derivative-free optimization. The method is specialized to Bernoulli mixed models with multivariate, correlated Gaussian random effects; here computations are performed using an inverse log-Cholesky parameterization of the Gaussian density that involves no matrix decomposition during model fitting, while Wald confidence intervals are provided for variance parameters on the original scale. Simulations give evidence of these intervals attaining nominal coverage if enough quadrature points are used, for data comprised of a large number of very small groups exhibiting large between-group heterogeneity. In contrast, the Laplace approximation is shown to give especially poor coverage and high bias for data comprised of a large number of small groups. Adaptive quadrature mitigates this, and the methods in this paper improve the computational feasibility of this more accurate method. All results may be reproduced using code available at \url{<a class="link-external link-https" href="https://github.com/awstringer1/aghmm-paper-code" rel="external noopener nofollow">this https URL</a>}.

Methodology,Computation
Sampling-Based Approaches to Calculating Marginal Densities

Alan E. Gelfand,Adrian F. M. Smith

DOI: https://doi.org/10.1080/01621459.1990.10476213

IF: 4.369

1990-06-01

Journal of the American Statistical Association

Abstract:Stochastic substitution, the Gibbs sampler, and the sampling-importance-resampling algorithm can be viewed as three alternative sampling- (or Monte Carlo-) based approaches to the calculation of numerical estimates of marginal probability distributions. The three approaches will be reviewed, compared, and contrasted in relation to various joint probability structures frequently encountered in applications. In particular, the relevance of the approaches to calculating Bayesian posterior densities for a variety of structured models will be discussed and illustrated.

statistics & probability
Efficient Approximations for the Marginal Likelihood of Incomplete Data Given a Bayesian Network

David Maxwell Chickering,David Heckerman

DOI: https://doi.org/10.48550/arXiv.1302.3567

2015-05-17

Abstract:We discuss Bayesian methods for learning Bayesian networks when data sets are incomplete. In particular, we examine asymptotic approximations for the marginal likelihood of incomplete data given a Bayesian network. We consider the Laplace approximation and the less accurate but more efficient BIC/MDL approximation. We also consider approximations proposed by Draper (1993) and Cheeseman and Stutz (1995). These approximations are as efficient as BIC/MDL, but their accuracy has not been studied in any depth. We compare the accuracy of these approximations under the assumption that the Laplace approximation is the most accurate. In experiments using synthetic data generated from discrete naive-Bayes models having a hidden root node, we find that the CS measure is the most accurate.

Machine Learning,Artificial Intelligence
On the marginal likelihood and cross-validation

E Fong,C C Holmes

DOI: https://doi.org/10.1093/biomet/asz077

IF: 3.0279

2020-01-24

Biometrika

Abstract:Summary In Bayesian statistics, the marginal likelihood, also known as the evidence, is used to evaluate model fit as it quantifies the joint probability of the data under the prior. In contrast, non-Bayesian models are typically compared using cross-validation on held-out data, either through $k$-fold partitioning or leave-$p$-out subsampling. We show that the marginal likelihood is formally equivalent to exhaustive leave-$p$-out crossvalidation averaged over all values of $p$ and all held-out test sets when using the log posterior predictive probability as the scoring rule. Moreover, the log posterior predictive score is the only coherent scoring rule under data exchangeability. This offers new insight into the marginal likelihood and cross-validation, and highlights the potential sensitivity of the marginal likelihood to the choice of the prior. We suggest an alternative approach using cumulative cross-validation following a preparatory training phase. Our work has connections to prequential analysis and intrinsic Bayes factors, but is motivated in a different way.

statistics & probability,mathematical & computational biology,biology
Efficient implementation of Markov chain Monte Carlo when using an unbiased likelihood estimator

Arnaud Doucet,Michael Pitt,George Deligiannidis,Robert Kohn

DOI: https://doi.org/10.48550/arXiv.1210.1871

2014-09-13

Abstract:When an unbiased estimator of the likelihood is used within a Metropolis--Hastings chain, it is necessary to trade off the number of Monte Carlo samples used to construct this estimator against the asymptotic variances of averages computed under this chain. Many Monte Carlo samples will typically result in Metropolis--Hastings averages with lower asymptotic variances than the corresponding Metropolis--Hastings averages using fewer samples. However, the computing time required to construct the likelihood estimator increases with the number of Monte Carlo samples. Under the assumption that the distribution of the additive noise introduced by the log-likelihood estimator is Gaussian with variance inversely proportional to the number of Monte Carlo samples and independent of the parameter value at which it is evaluated, we provide guidelines on the number of samples to select. We demonstrate our results by considering a stochastic volatility model applied to stock index returns.

Methodology
Marginal expected shortfall inference under multivariate regular variation

Simone A. Padoan,Stefano Rizzelli,Matteo Schiavone

2023-04-15

Abstract:Marginal expected shortfall is unquestionably one of the most popular systemic risk measures. Studying its extreme behaviour is particularly relevant for risk protection against severe global financial market downturns. In this context, results of statistical inference rely on the bivariate extreme values approach, disregarding the extremal dependence among a large number of financial institutions that make up the market. In order to take it into account we propose an inferential procedure based on the multivariate regular variation theory. We derive an approximating formula for the extreme marginal expected shortfall and obtain from it an estimator and its bias-corrected version. Then, we show their asymptotic normality, which allows in turn the confidence intervals derivation. Simulations show that the new estimators greatly improve upon the performance of existing ones and confidence intervals are very accurate. An application to financial returns shows the utility of the proposed inferential procedure. Statistical results are extended to a general $\beta$-mixing context that allows to work with popular time series models with heavy-tailed innovations.

Statistics Theory,Methodology
On the performance of some new Liu parameters for the gamma regression model

Muhammad Qasim,Muhammad Amin,Muhammad Amanullah

DOI: https://doi.org/10.1080/00949655.2018.1498502

IF: 1.225

2018-07-17

Journal of Statistical Computation and Simulation

Abstract:The maximum likelihood (ML) method is used to estimate the unknown Gamma regression (GR) coefficients. In the presence of multicollinearity, the variance of the ML method becomes overstated and the inference based on the ML method may not be trustworthy. To combat multicollinearity, the Liu estimator has been used. In this estimator, estimation of the Liu parameter d is an important problem. A few estimation methods are available in the literature for estimating such a parameter. This study has considered some of these methods and also proposed some new methods for estimation of the d. The Monte Carlo simulation study has been conducted to assess the performance of the proposed methods where the mean squared error (MSE) is considered as a performance criterion. Based on the Monte Carlo simulation and application results, it is shown that the Liu estimator is always superior to the ML and recommendation about which best Liu parameter should be used in the Liu estimator for the GR model is given.

statistics & probability,computer science, interdisciplinary applications
New results on the asymptotic and finite sample properties of the MaCML approach to multinomial probit model estimation

Manuel Batram,Dietmar Bauer

DOI: https://doi.org/10.48550/arXiv.1609.03295

2016-09-14

Abstract:In this paper the properties of the maximum approximate composite marginal likelihood (MaCML) approach to the estimation of multinomial probit models (MNP) proposed by Chandra Bhat and coworkers is investigated in finite samples as well as with respect to asymptotic properties. Using a small illustration example it is proven that the approach does not necessarily lead to consistent estimators for four different types of approximation of the Gaussian cumulative distribution function (including the Solow-Joe approach proposed by Bhat). It is shown that the bias of parameter estimates can be substantial (while typically it is small) and the bias in the corresponding implied probabilities is small but non-negligible. Furthermore in finite sample it is demonstrated by simulation that between two versions of the Solow-Joe method and two versions of the Mendell-Elston approximation no method dominates the others in terms of accuracy and numerical speed. Moreover the system to be estimated, the ordering of the components in the approximation method and even the tolerance used for stopping the numerical optimization routine all have an influence on the relative performance of the procedures corresponding to the various approximation methods. Jointly the paper thus points towards eminent research needs in order to decide on the method to use for a particular estimation problem at hand.

Methodology,Computation
Modified method of moments for generalized Laplace distributions

Adrian Fischer,Robert E. Gaunt,Andrey Sarantsev

DOI: https://doi.org/10.1080/03610918.2024.2307463

2024-03-07

Communications in Statistics - Simulation and Computation

Abstract:In this note, we consider the performance of the classic method of moments for parameter estimation of symmetric variance-gamma (generalized Laplace) distributions. We do this through both theoretical analysis (multivariate delta method) and a comprehensive simulation study with comparison to maximum likelihood estimation, finding performance is often unsatisfactory. In addition, we modify the method of moments by taking absolute moments to improve efficiency; in particular, our simulation studies demonstrate that our modified estimators have significantly improved performance for parameter values typically encountered in financial modeling, and is also competitive with maximum likelihood estimation.

statistics & probability
Marginal Likelihood Estimation for the Negative Binomial INGARCH Model

Jian Pei,Fukang Zhu

DOI: https://doi.org/10.1080/03610918.2022.2057542

2022-01-01

Abstract:In recent years, there has been increased interest in modeling integer-valued time series. Many methods for time series of counts have been developed in the literature because of their wide applications to epidemiology, finance, disease modeling and environmental science. The negative binomial integer-valued generalized autoregressive conditional heteroscedasticity model is a popular one, which can deal with both over-dispersion and potential extreme observations. The accurate estimation of the parameters in the model is extremely important. We adopt the marginal likelihood to estimate the intercept parameter and maximum likelihood to estimate other parameters of the model. We conduct simulations to assess the performance of this estimation method, and compare it with that of estimating all model parameters by maximum likelihood. The results show the superiority of proposed estimation method. We use two real examples to illustrate the model's ability to fit over-dispersed data and the validity of the estimation method.
Determine the Number of States in Hidden Markov Models via Marginal Likelihood

Yang Chen,Cheng-Der Fuh,Chu-Lan Michael Kao

2024-07-17

Abstract:Hidden Markov models (HMM) have been widely used by scientists to model stochastic systems: the underlying process is a discrete Markov chain and the observations are noisy realizations of the underlying process. Determining the number of hidden states for an HMM is a model selection problem, which is yet to be satisfactorily solved, especially for the popular Gaussian HMM with heterogeneous covariance. In this paper, we propose a consistent method for determining the number of hidden states of HMM based on the marginal likelihood, which is obtained by integrating out both the parameters and hidden states. Moreover, we show that the model selection problem of HMM includes the order selection problem of finite mixture models as a special case. We give rigorous proof of the consistency of the proposed marginal likelihood method and provide an efficient computation method for practical implementation. We numerically compare the proposed method with the Bayesian information criterion (BIC), demonstrating the effectiveness of the proposed marginal likelihood method.

Statistics Theory,Methodology
Skewed Bernstein-von Mises theorem and skew-modal approximations

Daniele Durante,Francesco Pozza,Botond Szabo

2024-04-08

Abstract:Gaussian approximations are routinely employed in Bayesian statistics to ease inference when the target posterior is intractable. Although these approximations are asymptotically justified by Bernstein-von Mises type results, in practice the expected Gaussian behavior may poorly represent the shape of the posterior, thus affecting approximation accuracy. Motivated by these considerations, we derive an improved class of closed-form approximations of posterior distributions which arise from a new treatment of a third-order version of the Laplace method yielding approximations in a tractable family of skew-symmetric distributions. Under general assumptions which account for misspecified models and non-i.i.d. settings, this family of approximations is shown to have a total variation distance from the target posterior whose rate of convergence improves by at least one order of magnitude the one established by the classical Bernstein-von Mises theorem. Specializing this result to the case of regular parametric models shows that the same improvement in approximation accuracy can be also derived for polynomially bounded posterior functionals. Unlike other higher-order approximations, our results prove that it is possible to derive closed-form and valid densities which are expected to provide, in practice, a more accurate, yet similarly-tractable, alternative to Gaussian approximations of the target posterior, while inheriting its limiting frequentist properties. We strengthen such arguments by developing a practical skew-modal approximation for both joint and marginal posteriors that achieves the same theoretical guarantees of its theoretical counterpart by replacing the unknown model parameters with the corresponding MAP estimate. Empirical studies confirm that our theoretical results closely match the remarkable performance observed in practice, even in finite, possibly small, sample regimes.

Statistics Theory,Computation,Methodology

Marginal Likelihood Calculation for the Gelfand–Dey and Chib Methods

Monte Carlo Approximation of Bayes Factors via Mixing with Surrogate Distributions

Marginal Likelihood Computation for Model Selection and Hypothesis Testing: An Extensive Review

Poisson and Gamma Model Marginalisation and Marginal Likelihood calculation using Moment-generating Functions

Improved marginal likelihood estimation via power posteriors and importance sampling

Identifying Structural Breaks by Marginal Likelihood

Lindeberg's Central Limit Theorems for Martingale-Like Sequences under Sub-Linear Expectations

Non-Gaussian Quasi Maximum Likelihood Estimation of GARCH Models

Variance Reduction for Generalized Likelihood Ratio Method by Conditional Monte Carlo and Randomized Quasi-Monte Carlo Methods

Approximate Marginal Likelihood Inference in Mixed Models for Grouped Data

Sampling-Based Approaches to Calculating Marginal Densities

Efficient Approximations for the Marginal Likelihood of Incomplete Data Given a Bayesian Network

On the marginal likelihood and cross-validation

Efficient implementation of Markov chain Monte Carlo when using an unbiased likelihood estimator

Marginal expected shortfall inference under multivariate regular variation

On the performance of some new Liu parameters for the gamma regression model

New results on the asymptotic and finite sample properties of the MaCML approach to multinomial probit model estimation

Modified method of moments for generalized Laplace distributions

Marginal Likelihood Estimation for the Negative Binomial INGARCH Model

Determine the Number of States in Hidden Markov Models via Marginal Likelihood

Skewed Bernstein-von Mises theorem and skew-modal approximations