Reduction of Restricted Maximum Likelihood for Random Coefficient Models

Kurt S. Riedel

DOI: https://doi.org/10.48550/arXiv.1803.03895

2018-03-11

Methodology

Abstract:The restricted maximum likelihood (REML) estimator of the dispersion matrix for random coefficient models is rewritten in terms of the sufficient statistics of the individual regressions.

What problem does this paper attempt to address?

Maximum Likelihood Estimators in a Two Step Model for PLS

Ying Li,Dietrich von Rosen

DOI: https://doi.org/10.1080/03610926.2011.607531

2012-01-01

Abstract:Univariate partial least squares regression (PLS1) is a method of modeling relationships between a response variable and explanatory variables, especially when the explanatory variables are almost collinear. The purpose is to predict a future response observation, although in many applications there is an interest to understand the contributions of each explanatory variable. It is an algorithmic approach. In this article, we are going to use the algorithm presented by Helland (1988). The population PLS predictor is linked to a linear model including a Krylov design matrix and a two-step estimation procedure. For the first step, the maximum likelihood approach is applied to a specific multivariate linear model, generating tools for evaluating the information in the explanatory variables. It is shown that explicit maximum likelihood estimators of the dispersion matrix can be obtained where the dispersion matrix, besides representing the variation in the error, also includes the Krylov structured design matrix describing the mean.
Restricted maximum likelihood estimation in generalized linear mixed models

Luca Maestrini,Francis K.C. Hui,Alan H. Welsh

2024-02-20

Abstract:Restricted maximum likelihood (REML) estimation is a widely accepted and frequently used method for fitting linear mixed models, with its principal advantage being that it produces less biased estimates of the variance components. However, the concept of REML does not immediately generalize to the setting of non-normally distributed responses, and it is not always clear the extent to which, either asymptotically or in finite samples, such generalizations reduce the bias of variance component estimates compared to standard unrestricted maximum likelihood estimation. In this article, we review various attempts that have been made over the past four decades to extend REML estimation in generalized linear mixed models. We establish four major classes of approaches, namely approximate linearization, integrated likelihood, modified profile likelihoods, and direct bias correction of the score function, and show that while these four classes may have differing motivations and derivations, they often arrive at a similar if not the same REML estimate. We compare the finite sample performance of these four classes through a numerical study involving binary and count data, with results demonstrating that they perform similarly well in reducing the finite sample bias of variance components.

Methodology,Statistics Theory
Regularized maximum likelihood estimation for the random coefficients model

Fabian Dunker Emil Mendoza Marco Reale University of Canterbury,School of Mathematics and Statistics,University of Canterbury,Christchurch,New Zealand

DOI: https://doi.org/10.1080/07474938.2024.2410765

IF: 1.2

2024-11-07

Econometric Reviews

Abstract:The random coefficients model Yi=β0i+β1iX1i+β2iX2i+...+βdiXdi , with X i , Y i , β i i.i.d, and β i independent of X i is often used to capture unobserved heterogeneity in a population. We propose a quasi-maximum likelihood method to estimate the joint density distribution of the random coefficient model. This method implicitly involves the inversion of the Radon transformation in order to reconstruct the joint distribution, and hence is an inverse problem. To add stability to the solution, we apply Tikhonov-type regularization methods. Nonparametric estimation for the joint density of βi=(β0i,...,βdi) based on kernel methods or Fourier inversion have been proposed in recent years. Most of these methods assume a heavy tailed design density f X . We analyze the convergence of the quasi maximum likelihood method without assuming heavy tails for f X and illustrate performance by applying the method on simulated and real data.

economics,social sciences, mathematical methods,mathematics, interdisciplinary applications,statistics & probability
Decorrelated Empirical Likelihood for Generalized Linear Models with High-Dimensional Longitudinal Data

Shuli Geng,Lixin Zhang

DOI: https://doi.org/10.1016/j.spl.2024.110135

IF: 0.718

2024-01-01

Statistics & Probability Letters

Abstract:This paper focuses on decorrelated empirical likelihood-based inference for longitudinal data with ultrahigh-dimensional covariates. The primary issues we aim to address involve parameter estimation and hypothesis testing for a low-dimensional parameter of interest. Under the framework of the generalized linear model, we initially consider the within-subject correlation by linearizing the precision matrix with certain known matrices, which retains optimality even if the working correlated structure is misspecified. Coupled with the decorrelated matrix, we then eliminate the influence of nuisance parameters on the estimation procedure. The proposed approach not only yields more efficient estimators compared to generalized decorrelated estimating equations but also shares the same asymptotic variance as quadratic decorrelated inference function based methods. Furthermore, we define the decorrelated empirical log-likelihood ratio test statistic to assess the significance of regression coefficients. Finally, to evaluate the performance of the proposed procedure, we conduct simulation studies and apply it to a real data example.
Computing log-likelihood and its derivatives for restricted maximum likelihood methods

Shengxin Zhu

DOI: https://doi.org/10.48550/arXiv.1608.07207

2016-08-25

Numerical Analysis

Abstract:Recent large scale genome wide association analysis involves large scale linear mixed models. Quantifying (co)-variance parameters in the mixed models with a restricted maximum likelihood method results in a score function which is the first derivative of a log-likelihood. To obtain a statistically efficient estimate of the variance parameters, one needs to find the root of the score function via the Newton method. Most elements of the Jacobian matrix of the score involve a trace term of four parametric matrix-matrix multiplications. It is computationally prohibitively for large scale data sets. By a serial matrix transforms and an averaged information splitting technique, an approximate Jacobian matrix can be obtained by splitting the average of the Jacobian matrix and its expected value. In the approximated Jacobian, its elements only involve Four matrix vector multiplications which can be efficiently evaluated by solving sparse linear systems with the multi-frontal factorization method.
Selection of Regression Models under Linear Restrictions for Fixed and Random Designs

Sen Tian,Clifford M. Hurvich,Jeffrey S. Simonoff

DOI: https://doi.org/10.48550/arXiv.2009.10029

2020-09-22

Abstract:Many important modeling tasks in linear regression, including variable selection (in which slopes of some predictors are set equal to zero) and simplified models based on sums or differences of predictors (in which slopes of those predictors are set equal to each other, or the negative of each other, respectively), can be viewed as being based on imposing linear restrictions on regression parameters. In this paper, we discuss how such models can be compared using information criteria designed to estimate predictive measures like squared error and Kullback-Leibler (KL) discrepancy, in the presence of either deterministic predictors (fixed-X) or random predictors (random-X). We extend the justifications for existing fixed-X criteria Cp, FPE and AICc, and random-X criteria Sp and RCp, to general linear restrictions. We further propose and justify a KL-based criterion, RAICc, under random-X for variable selection and general linear restrictions. We show in simulations that the use of the KL-based criteria AICc and RAICc results in better predictive performance and sparser solutions than the use of squared error-based criteria, including cross-validation.

Methodology
Sparse maximum likelihood estimation for regression models

Min Tsao

2024-04-08

Abstract:For regression model selection via maximum likelihood estimation, we adopt a vector representation of candidate models and study the likelihood ratio confidence region for the regression parameter vector of a full model. We show that when its confidence level increases with the sample size at a certain speed, with probability tending to one, the confidence region consists of vectors representing models containing all active variables, including the true parameter vector of the full model. Using this result, we examine the asymptotic composition of models of maximum likelihood and find the subset of such models that contain all active variables. We then devise a consistent model selection criterion which has a sparse maximum likelihood estimation interpretation and certain advantages over popular information criteria.

Statistics Theory
Essential formulae for restricted maximum likelihood and its derivatives associated with the linear mixed models

Shengxin Zhu,Andrew J Wathen

DOI: https://doi.org/10.48550/arXiv.1805.05188

2018-05-11

Computation

Abstract:The restricted maximum likelihood method enhances popularity of maximum likelihood methods for variance component analysis on large scale unbalanced data. As the high throughput biological data sets and the emerged science on uncertainty quantification, such a method receives increasing attention. Estimating the unknown variance parameters with restricted maximum likelihood method usually requires an nonlinear iterative method. Therefore proper formulae for the log-likelihood function and its derivatives play an essential role in practical algorithm design. It is our aim to provide a mathematical introduction to this method, and supply a self-contained derivation on some available formulae used in practical algorithms. Some new proof are supplied.
Stochastic Model Reduction by Maximizing Independence

Hui Zhang,You‐Xian Sun

DOI: https://doi.org/10.1002/apj.5500130302

2005-01-01

Developments in Chemical Engineering and Mineral Processing

Abstract:AbstractBy analysing information descriptions in state space models of linear stochastic systems, this paper proposes two model reduction methods via principles of maximizing independence and conditional independence among the reduced state vector, respectively. These methods are based on state aggregation. The independence and conditional independence are measured by the Kullback‐Leibler information distance. It is demonstrated that the maximum conditional independence method is not only applicable to stable systems, but also applicable to unstable systems. Simulation results illustrate the efficiency of the present methods.
Iteratively Reweighted Least Squares Method for Estimating Polyserial and Polychoric Correlation Coefficients

Peng Zhang,Ben Liu,Jingjing Pan

DOI: https://doi.org/10.1080/10618600.2023.2257251

2024-01-01

Journal of Computational and Graphical Statistics

Abstract:An iteratively reweighted least squares (IRLS) method is proposed for estimating polyserial and polychoric correlation coefficients in this paper. It iteratively calculates the slopes in a series of weighted linear regression models fitting on conditional expected values. For polyserial correlation coefficient, conditional expectations of the latent predictor is derived from the observed ordinal categorical variable, and the regression coefficient is obtained using weighted least squares method. In estimating polychoric correlation coefficient, conditional expectations of the response variable and the predictor are updated in turns. Standard errors of the estimators are obtained using the delta method based on data summaries instead of the whole data. Conditional univariate normal distribution is exploited and a single integral is numerically evaluated in the proposed algorithm, comparing to the double integral computed numerically based on the bivariate normal distribution in the traditional maximum likelihood (ML) approaches. This renders the new algorithm very fast in estimating both polyserial and polychoric correlation coefficients. Thorough simulation studies are conducted to compare the performances of the proposed method with the classical ML methods. Real data analyses illustrate the advantage of the new method in computation speed.
On the maximum likelihood estimation in general log-linear models

Anna Klimova,Matthias Kuhn

DOI: https://doi.org/10.48550/arXiv.2208.12024

2022-08-25

Methodology

Abstract:General log-linear models specified by non-negative integer design matrices have a potentially wide range of applications, although using models without the genuine overall effect, that is, ones which cannot be reparameterized to include a normalizing constant, is still rare. The log-linear models without the overall effect arise naturally in practice, and can be handled in a similar manner to models with the overall effect. A novel iterative scaling procedure for the MLE computation under such models is proposed, and its convergence is proved. The results are illustrated using data from a recent clinical study.
Nonparametric maximum likelihood estimation under a likelihood ratio order

Ted Westling,Kevin J. Downes,Dylan S. Small

DOI: https://doi.org/10.48550/arXiv.1904.12321

2021-07-07

Abstract:Comparison of two univariate distributions based on independent samples from them is a fundamental problem in statistics, with applications in a wide variety of scientific disciplines. In many situations, we might hypothesize that the two distributions are stochastically ordered, meaning intuitively that samples from one distribution tend to be larger than those from the other. One type of stochastic order that arises in economics, biomedicine, and elsewhere is the likelihood ratio order, also known as the density ratio order, in which the ratio of the density functions of the two distributions is monotone non-decreasing. In this article, we derive and study the nonparametric maximum likelihood estimator of the individual distributions and the ratio of their densities under the likelihood ratio order. Our work applies to discrete distributions, continuous distributions, and mixed continuous-discrete distributions. We demonstrate convergence in distribution of the estimator in certain cases, and we illustrate our results using numerical experiments and an analysis of a biomarker for predicting bacterial infection in children with systemic inflammatory response syndrome.

Methodology,Statistics Theory
Maximum Likelihood for Matrices with Rank Constraints

Jonathan Hauenstein,Jose Rodriguez,Bernd Sturmfels

DOI: https://doi.org/10.48550/arXiv.1210.0198

2013-03-18

Abstract:Maximum likelihood estimation is a fundamental optimization problem in statistics. We study this problem on manifolds of matrices with bounded rank. These represent mixtures of distributions of two independent discrete random variables. We determine the maximum likelihood degree for a range of determinantal varieties, and we apply numerical algebraic geometry to compute all critical points of their likelihood functions. This led to the discovery of maximum likelihood duality between matrices of complementary ranks, a result proved subsequently by Draisma and Rodriguez.

Algebraic Geometry,Optimization and Control,Computation
Maximum pseudo-likelihood estimation in copula models for small weakly dependent samples

Alexandra Dias

DOI: https://doi.org/10.48550/arXiv.2208.01322

2022-09-06

Abstract:Maximum pseudo-likelihood (MPL) is a semiparametric estimation method often used to obtain the dependence parameters in copula models from data. It has been shown that despite being consistent, and in some cases efficient, MPL estimation can overestimate the level of dependence especially for small weakly dependent samples. We show that the MPL method uses the expected value of order statistics and we propose to use instead the median or the mode of the same order statistics. In a simulation study we compare the finite-sample performance of the proposed estimators with that of the original MPL and the inversion method estimators based on Kendall's tau and Spearman's rho. Our results indicate that the modified MPL estimators, especially the one based on the mode of the order statistics, have better finite-sample performance, while still enjoying the large-sample properties of the original MPL method.

Methodology
A Note on an R^2 Measure for Fixed Effects in the Generalized Linear Mixed Model

Lloyd J. Edwards

DOI: https://doi.org/10.48550/arXiv.1011.0471

2010-11-02

Abstract:Using the LRT statistic, a model R^2 is proposed for the generalized linear mixed model for assessing the association between the correlated outcomes and fixed effects. The R^2 compares the full model to a null model with all fixed effects deleted.

Methodology,Applications
Misspecification Analysis of High-Dimensional Random Effects Models for Estimation of Signal-to-Noise Ratios

Xiaohan Hu,Xiaodong Li

2023-06-08

Abstract:Estimation of signal-to-noise ratios and residual variances in high-dimensional linear models has various important applications including, e.g. heritability estimation in bioinformatics. One commonly used estimator, usually referred to as REML, is based on the likelihood of the random effects model, in which both the regression coefficients and the noise variables are respectively assumed to be i.i.d Gaussian random variables. In this paper, we aim to establish the consistency and asymptotic distribution of the REML estimator for the SNR, when the actual coefficient vector is fixed, and the actual noise is heteroscedastic and correlated, at the cost of assuming the entries of the design matrix are independent and skew-free. The asymptotic variance can be also consistently estimated when the noise is heteroscedastic but uncorrelated. Extensive numerical simulations illustrate our theoretical findings and also suggest some assumptions imposed in our theoretical results are likely relaxable.

Statistics Theory,Methodology
Rational Maximum Likelihood Estimators of Kronecker Covariance Matrices

Mathias Drton,Alexandros Grosdos,Andrew McCormack

2024-06-02

Abstract:As is the case for many curved exponential families, the computation of maximum likelihood estimates in a multivariate normal model with a Kronecker covariance structure is typically carried out with an iterative algorithm, specifically, a block-coordinate ascent algorithm. In this article we highlight a setting, specified by a coprime relationship between the sample size and dimension of the Kronecker factors, where the likelihood equations have algebraic degree one and an explicit, easy-to-evaluate rational formula for the maximum likelihood estimator can be found. A partial converse of this result is provided that shows that outside of the aforementioned special setting and for large sample sizes, examples of data sets can be constructed for which the degree of the likelihood equations is larger than one.

Statistics Theory
Small Sample Methods for Multilevel Modeling: A Colloquial Elucidation of REML and the Kenward-Roger Correction

Daniel McNeish

DOI: https://doi.org/10.1080/00273171.2017.1344538

2017-07-17

Multivariate Behavioral Research

Abstract:Studies on small sample properties of multilevel models have become increasingly prominent in the methodological literature in response to the frequency with which small sample data appear in empirical studies. Simulation results generally recommend that empirical researchers employ restricted maximum likelihood estimation (REML) with a Kenward-Roger correction with small samples in frequentist contexts to minimize small sample bias in estimation and to prevent inflation of Type-I error rates. However, simulation studies focus on recommendations for best practice, and there is little to no explanation of why traditional maximum likelihood (ML) breaks down with smaller samples, what differentiates REML from ML, or how the Kenward-Roger correction remedies lingering small sample issues. Due to the complexity of these methods, most extant descriptions are highly mathematical and are intended to prove that the methods improve small sample performance as intended. Thus, empirical researchers have documentation that these methods are advantageous but still lack resources to help understand what the methods actually do and why they are needed. This tutorial explains why ML falters with small samples, how REML circumvents some issues, and how Kenward-Roger works. We do so without equations or derivations to support more widespread understanding and use of these valuable methods.

psychology, experimental,mathematics, interdisciplinary applications,statistics & probability,social sciences, mathematical methods
Maximum Likelihood Inference in Robust Linear Mixed-Effects Models Using Multivariate T Distributions

Peter X. -K. Song,Peng Zhang,Annie Qu

IF: 1.4

2007-01-01

Statistica Sinica

Abstract:This paper focuses on the problem of maximum likelihood estimation in linear mixed-effects models where outliers or unduly large observations are present in clustered or longitudinal data. Multivariate t distributions are often imposed on either random effects and/or random errors to incorporate outliers. A powerful algorithm of maximum by parts (MBP) proposed by Song, Fan and Kalbfleisch (2005) is implemented to obtain maximum likelihood estimators when the likelihood is intractable. The computational efficiency of the MBP allows us to further apply a profile-likelihood technique for the estimation of the degrees of freedom in t-distributions. Comparison of the Akaike information criterion (AIC) among candidate models provides an objective criterion to determine whether outliers are influential on the quality of model fit. The proposed models and methods are illustrated through both simulation studies and data analysis examples, with comparison to the existing EM-algorithm.
Newton-type Methods for REML Estimation in Genetic Analysis of Quantitative Traits

Kateryna Mishchenko,Sverker Holmgren,Lars Ronnegard

DOI: https://doi.org/10.48550/arXiv.0711.2619

2007-11-16

Abstract:Robust and efficient optimization methods for variance component estimation using Restricted Maximum Likelihood (REML) models for genetic mapping of quantitative traits are considered. We show that the standard Newton-AI scheme may fail when the optimum is located at one of the constraint boundaries, and we introduce different approaches to remedy this by taking the constraints into account. We approximate the Hessian of the objective function using the average information matrix and also by using an inverse BFGS formula. The robustness and efficiency is evaluated for problems derived from two experimental data from the same animal populations.

Other Quantitative Biology,Quantitative Methods

Reduction of Restricted Maximum Likelihood for Random Coefficient Models

Maximum Likelihood Estimators in a Two Step Model for PLS

Restricted maximum likelihood estimation in generalized linear mixed models

Regularized maximum likelihood estimation for the random coefficients model

Decorrelated Empirical Likelihood for Generalized Linear Models with High-Dimensional Longitudinal Data

Computing log-likelihood and its derivatives for restricted maximum likelihood methods

Selection of Regression Models under Linear Restrictions for Fixed and Random Designs

Sparse maximum likelihood estimation for regression models

Essential formulae for restricted maximum likelihood and its derivatives associated with the linear mixed models

Stochastic Model Reduction by Maximizing Independence

Iteratively Reweighted Least Squares Method for Estimating Polyserial and Polychoric Correlation Coefficients

On the maximum likelihood estimation in general log-linear models

Nonparametric maximum likelihood estimation under a likelihood ratio order

Maximum Likelihood for Matrices with Rank Constraints

Maximum pseudo-likelihood estimation in copula models for small weakly dependent samples

A Note on an R^2 Measure for Fixed Effects in the Generalized Linear Mixed Model

Misspecification Analysis of High-Dimensional Random Effects Models for Estimation of Signal-to-Noise Ratios

Rational Maximum Likelihood Estimators of Kronecker Covariance Matrices

Small Sample Methods for Multilevel Modeling: A Colloquial Elucidation of REML and the Kenward-Roger Correction

Maximum Likelihood Inference in Robust Linear Mixed-Effects Models Using Multivariate T Distributions

Newton-type Methods for REML Estimation in Genetic Analysis of Quantitative Traits