Correlation Estimation System Minimization Compared to Least Squares Minimization in Simple Linear Regression

Rudy Gideon

DOI: https://doi.org/10.48550/arXiv.1802.02649

2018-02-08

Abstract:A general method of minimization using correlation coefficients and order statistics is evaluated relative to least squares procedures in the estimation of parameters for normal data in simple linear regression.

Methodology

What problem does this paper attempt to address?

Iteratively Reweighted Least Squares Method for Estimating Polyserial and Polychoric Correlation Coefficients

Peng Zhang,Ben Liu,Jingjing Pan

DOI: https://doi.org/10.1080/10618600.2023.2257251

2024-01-01

Journal of Computational and Graphical Statistics

Abstract:An iteratively reweighted least squares (IRLS) method is proposed for estimating polyserial and polychoric correlation coefficients in this paper. It iteratively calculates the slopes in a series of weighted linear regression models fitting on conditional expected values. For polyserial correlation coefficient, conditional expectations of the latent predictor is derived from the observed ordinal categorical variable, and the regression coefficient is obtained using weighted least squares method. In estimating polychoric correlation coefficient, conditional expectations of the response variable and the predictor are updated in turns. Standard errors of the estimators are obtained using the delta method based on data summaries instead of the whole data. Conditional univariate normal distribution is exploited and a single integral is numerically evaluated in the proposed algorithm, comparing to the double integral computed numerically based on the bivariate normal distribution in the traditional maximum likelihood (ML) approaches. This renders the new algorithm very fast in estimating both polyserial and polychoric correlation coefficients. Thorough simulation studies are conducted to compare the performances of the proposed method with the classical ML methods. Real data analyses illustrate the advantage of the new method in computation speed.
On the minimax robustness against correlation and heteroscedasticity of ordinary least squares among generalized least squares estimators of regression

Douglas P. Wiens

2024-04-02

Abstract:We present a result according to which certain functions of covariance matrices are maximized at scalar multiples of the identity matrix. In a statistical context in which such functions measure loss, this says that the least favourable form of dependence is in fact independence, so that a procedure optimal for i.i.d.\ data can be minimax. In particular, the ordinary least squares (\textsc{ols}) estimate of a correctly specified regression response is minimax among generalized least squares (\textsc{gls}) estimates, when the maximum is taken over certain classes of error covariance structures and the loss function possesses a natural monotonicity property. An implication is that it can be not only safe, but optimal to ignore such departures from the usual assumption of i.i.d.\ errors. We then consider regression models in which the response function is possibly misspecified, and show that \textsc{ols} is minimax if the design is uniform on its support, but that this often fails otherwise. We go on to investigate the interplay between minimax \textsc{gls} procedures and minimax designs, leading us to extend, to robustness against dependencies, an existing observation -- that robustness against model misspecifications is increased by splitting replicates into clusters of observations at nearby locations.

Statistics Theory
Novel Robust Least-Squares Estimator for Linear Dynamic Data Reconciliation

Rui Zhang,Yi Zhang,Yiping Feng,Gang Rong

DOI: https://doi.org/10.1021/acs.iecr.8b04853

2018-01-01

Industrial & Engineering Chemistry Research

Abstract:Dynamic data reconciliation (DDR) is used to reduce the uncertainties in process measurement. Conventional data reconciliation theories and methods are based on least-squares estimation, whose conditions are hard to meet in real-world applications. Since least-squares estimators can be made robust by equivalent weight, many works concentrated on robust estimators and their performance in data reconciliation. However, robust least-squares algorithm is still unable to detect gross errors in low-redundancy variables. In this article, our work takes a step further in improving the robustness of the Huber estimator by taking structural redundancy into consideration. Modifications on influence function and weight function are made by involving local redundancy and rejection interval. The detectivity of gross error in low-redundant variables is increased. Combined with the above improvement and then taking advantage of online filtering, a novel robust dynamic data reconciliation algorithm is proposed. Case studi...
Predictive Regressions: Method of Least Autocorrelation

John Qi Zhu,Yan Liu

DOI: https://doi.org/10.2139/ssrn.928877

2006-01-01

SSRN Electronic Journal

Abstract:Conventional predictive regressions produce biased and inefficient coefficient estimates in small samples when the predicting variable is Gaussian first order persistent and its innovations are highly correlated with the error series of the return. We propose a new estimation method (the method of least autocorrelation) conditional on the assumption of serially uncorrelated innovations to the dependent variable. The new estimator minimizes the weighted sum of squared autocorrelations of residuals obtained from fitting a return series through estimated parameters. The asymptotic properties of the estimator and a refined weighting matrix are discussed. Finite sample behavior of the estimator is compared with existing estimation methods in the simulation study. The new method produces more accurate and efficient point estimate of the true parameter and improves performance in the power and size tests. Our empirical results provide some fresh evidence on the predictability of stock returns during the post-war time period.
Statistical Inference for a Robust Measure of Multiple Correlation

Catherine Dehon,Christophe Croux

DOI: https://doi.org/10.1007/978-3-642-57489-4_86

2002-01-01

Abstract:In regression analysis it is standard practice to report an R-squared statistic. This measure of multiple correlation is based on the Least Squares estimator, which is known to be extremely sensitive to outliers. The associated R-squared suffers from the same lack of robustness. Also the value of the F-statistic, used to see whether the explanatory variables have jointly a significant influence on the dependent variable, exhibits similar problems. Many robust estimation procedures are available now, but there has been less emphasis on the development of the inference part and in particular on the study of robust R-squared measures. It is however clear that applied statisticians desire to have the same tools to validate their model as when using the classical least squares estimator. In this note we will study the stability properties of robust measures of multiple correlation, and also investigate the size and power of F-statistics based on them.
Robust Estimating Equations and Bias Correction of Correlation Parameters for Longitudinal Data

Guo You Qin,Zhong Yi Zhu,Wing K. Fung

DOI: https://doi.org/10.1016/j.csda.2008.03.019

IF: 2.035

2008-01-01

Computational Statistics & Data Analysis

Abstract:The estimation of correlation parameters has received attention for both its own interest and improvement of the estimation efficiency of mean parameters by the generalized estimating equations (GEE) approach. Many of the well-established methods for the estimation of correlation parameters can be constructed under the GEE framework which is, however, sensitive to outliers. In this paper, we consider two ways of constructing robust estimating equations for achieving robust estimation of the correlation parameters. Furthermore, the estimators of the correlation parameters from the robustified GEE may be still biased as the expectation of the estimating equation is biased from zero when the underlying distribution is not symmetric. Therefore, bias-corrected robust estimators of correlation parameters are proposed. The performance of the proposed methods are investigated by simulation. The results show that the proposed robust and bias-corrected robust estimators can reduce the bias successfully. Two real data sets are analyzed for illustration.
Estimators of the correlation coefficient in the bivariate exponential distribution

W. J. Szajnowski

DOI: https://doi.org/10.48550/arXiv.1702.03080

2017-02-10

Abstract:A finite-support constraint on the parameter space is used to derive a lower bound on the error of an estimator of the correlation coefficient in the bivariate exponential distribution. The bound is then exploited to examine optimality of three estimators, each being a nonlinear function of moments of exponential or Rayleigh observables. The estimator based on a measure of cosine similarity is shown to be highly efficient for values of the correlation coefficient greater than 0.35; for smaller values, however, it is the transformed Pearson correlation coefficient that exhibits errors closer to the derived bound.

Methodology
Asymptotic Theory of Least Distances Estimate in Multivariate Linear Models

Z. D. Bai,X.R. Chen,B.Q. Miao,C. Radhakrishna Rao

DOI: https://doi.org/10.1080/02331889008802260

IF: 2.346

1990-01-01

Statistics

Abstract:We consider the multivariate linear model where Yi is a p-vector random variable, Xi is a q X p matrix, βA0 is an unknown q-vector parameter and {si} is a sequence of iid p-vector random variable with median vector zero. The estimate βAn of βA0 such that is called the least distances (LD) estimator. It may be recalled that the least squares (LS) estimator is obtained by minimizing the sum of norm squares. In this paper, it is shown that the LD estimator is unique, consistent and has an asymptotic q-variate normal distribution with mean β0 and co variance matrix V which depends on the distribution of the error vectors {ei}. A consistent estimator of V is proposed which together with βAn enables an asymptotic inference on β0. In particular, tests of linear hypotheses on β0 analogous to those of analysis of variance in the GATJSS-MARKOFF linear model are developed. Explicit expressions are obtained in some cases for the asymptotic relative efficiency of the LD compared to the LS estimator.
Robust Estimation of Polychoric Correlation

Max Welz,Patrick Mair,Andreas Alfons

2024-10-24

Abstract:Polychoric correlation is often an important building block in the analysis of rating data, particularly for structural equation models. However, the commonly employed maximum likelihood (ML) estimator is highly susceptible to misspecification of the polychoric correlation model, for instance through violations of latent normality assumptions. We propose a novel estimator that is designed to be robust to partial misspecification of the polychoric model, that is, the model is only misspecified for an unknown fraction of observations, for instance (but not limited to) careless respondents. In contrast to existing literature, our estimator makes no assumption on the type or degree of model misspecification. It furthermore generalizes ML estimation and is consistent as well as asymptotically normally distributed. We demonstrate the robustness and practical usefulness of our estimator in simulation studies and an empirical application on a Big Five administration. In the latter, the polychoric correlation estimates of our estimator and ML differ substantially, which, after further inspection, is likely due to the presence of careless respondents that the estimator helps identify.

Methodology,Statistics Theory,Applications,Other Statistics
Optimal low-rank approximation to a correlation matrix

Zhenyue Zhang,Lixin Wu

DOI: https://doi.org/10.1016/S0024-3795(02)00551-7

IF: 1.307

2003-01-01

Linear Algebra and its Applications

Abstract:Low-rank approximation of a correlation matrix is a constrained minimization problem that can be translated into a minimization–maximization problem by the method of Lagrange multiplier. In this paper, we solve the inner maximization problems with a single spectral decomposition, and the outer minimization problems with gradient-based descending methods. An in-depth analysis is done to characterize the solutions of the inner maximization problem for the case when they are non-unique. The well-posedness of the Lagrange multiplier problem and the convergence of the descending methods are rigorously justified. Numerical results are presented.
Partial Least Squares Methods: Partial Least Squares Correlation and Partial Least Square Regression

Hervé Abdi,Lynne J. Williams

DOI: https://doi.org/10.1007/978-1-62703-059-5_23

2012-08-18

Abstract:Partial least square (pls) methods (also sometimes called projection to latent structures) relate the information present in two data tables that collect measurements on the same set of observations. pls methods proceed by deriving latent variables which are (optimal) linear combinations of the variables of a data table. When the goal is to find the shared information between two tables, the approach is equivalent to a correlation problem and the technique is then called partial least square correlation (PLSC) (also sometimes called pls-SVD). In this case there are two sets of latent variables (one set per table), and these latent variables are required to have maximal covariance. When the goal is to predict one data table the other one, the technique is then called partial least square regression. In this case there is one set of latent variables (derived from the predictor table) and these latent variables are required to give the best possible prediction. In this paper we present and illustrate PLSC and PLSR and show how these descriptive multivariate analysis techniques can be extended to deal with inferential questions by using cross-validation techniques such as the bootstrap and permutation tests.
Kendall Correlation Coefficients for Portfolio Optimization

Tomas Espana,Victor Le Coz,Matteo Smerlak

2024-10-23

Abstract:Markowitz's optimal portfolio relies on the accurate estimation of correlations between asset returns, a difficult problem when the number of observations is not much larger than the number of assets. Using powerful results from random matrix theory, several schemes have been developed to "clean" the eigenvalues of empirical correlation matrices. By contrast, the (in practice equally important) problem of correctly estimating the eigenvectors of the correlation matrix has received comparatively little attention. Here we discuss a class of correlation estimators generalizing Kendall's rank correlation coefficient which improve the estimation of both eigenvalues and eigenvectors in data-poor regimes. Using both synthetic and real financial data, we show that these generalized correlation coefficients yield Markowitz portfolios with lower out-of-sample risk than those obtained with rotationally invariant estimators. Central to these results is a property shared by all Kendall-like estimators but not with classical correlation coefficients: zero eigenvalues only appear when the number of assets becomes proportional to the square of the number of data points.

Statistical Finance,Statistical Mechanics
A compromise criterion for weighted least squares estimates

Jordan Bryan,Haibo Zhou,Didong Li

2024-10-29

Abstract:In the heteroscedastic linear model, the weighted least squares estimate of the model coefficients is more efficient than the ordinary least squares estimate. However, the practical application of weighted least squares is challenging because it requires knowledge of the error variances. Feasible weighted least squares estimates, which use approximations of the variances when they are unknown, may either be more or less efficient than the ordinary least squares estimate depending on the quality of the approximation. A direct comparison between feasible and ordinary least squares has significant implications for the application of regression analysis in varied fields, yet such a comparison remains an unresolved challenge. In this study, we address this challenge by identifying the conditions under which feasible weighted least squares estimates using fixed weights demonstrate greater efficiency than the ordinary least squares estimate. These conditions provide guidance for the design of feasible estimates using random weights. They also shed light on how certain robust regression estimates behave with respect to the linear model with normal errors of unequal variance.

Statistics Theory,Methodology
Least Squares Inference for Data with Network Dependency

Jing Lei,Kehui Chen,Haeun Moon

2024-04-02

Abstract:We address the inference problem concerning regression coefficients in a classical linear regression model using least squares estimates. The analysis is conducted under circumstances where network dependency exists across units in the sample. Neglecting the dependency among observations may lead to biased estimation of the asymptotic variance and often inflates the Type I error in coefficient inference. In this paper, we first establish a central limit theorem for the ordinary least squares estimate, with a verifiable dependence condition alongside corresponding neighborhood growth conditions. Subsequently, we propose a consistent estimator for the asymptotic variance of the estimated coefficients, which employs a data-driven method to balance the bias-variance trade-off. We find that the optimal tuning depends on the linear hypothesis under consideration and must be chosen adaptively. The presented theory and methods are illustrated and supported by numerical experiments and a data example.

Methodology,Statistics Theory
Correlation Estimation in Hybrid Systems

Baron Law

DOI: https://doi.org/10.1142/S0219024923500085

2023-07-07

Abstract:A simple method is proposed to estimate the instantaneous correlations between state variables in a hybrid system from the empirical correlations between observable market quantities such as spot rate, stock price and implied volatility. The new algorithm is extremely fast since only low-dimension linear systems are involved. If the resulting matrix from the linear systems is not positive semidefinite, the shrinking method, which requires only bisection-style iterations, is recommended to convert the matrix to positive semidefinite. The square of short-term at-the-money implied volatility is suggested as the proxy for the unobservable stochastic variance. When the implied volatility is not available, a simple trick is provided to fill in the missing correlations. Numerical study shows that the estimates are reasonably accurate, when using more than 1,000 data points. In addition, the algorithm is robust to misspecified interest rate model parameters and the short-sampling-period assumption. G2++ and Heston are used for illustration but the method can be extended to other affine term structure, local volatility and jump diffusion models, with or without stochastic interest rate.

Computational Finance,Pricing of Securities
With random regressors, least squares inference is robust to correlated errors with unknown correlation structure

Zifeng Zhang,Peng Ding,Wen Zhou,Haonan Wang

2024-10-10

Abstract:Linear regression is arguably the most widely used statistical method. With fixed regressors and correlated errors, the conventional wisdom is to modify the variance-covariance estimator to accommodate the known correlation structure of the errors. We depart from the literature by showing that with random regressors, linear regression inference is robust to correlated errors with unknown correlation structure. The existing theoretical analyses for linear regression are no longer valid because even the asymptotic normality of the least-squares coefficients breaks down in this regime. We first prove the asymptotic normality of the t statistics by establishing their Berry-Esseen bounds based on a novel probabilistic analysis of self-normalized statistics. We then study the local power of the corresponding t tests and show that, perhaps surprisingly, error correlation can even enhance power in the regime of weak signals. Overall, our results show that linear regression is applicable more broadly than the conventional theory suggests, and further demonstrate the value of randomization to ensure robustness of inference.

Statistics Theory,Methodology
Asymptotic confidence interval for R2 in multiple linear regression

J Dedecker,O Guedj,M L Taupin

2024-01-23

Abstract:Following White's approach of robust multiple linear regression, we give asymptotic confidence intervals for the multiple correlation coefficient R2 under minimal moment conditions. We also give the asymptotic joint distribution of the empirical estimators of the individual R2's. Through different sets of simulations, we show that the procedure is indeed robust (contrary to the procedure involving the near exact distribution of the empirical estimator of R2 is the multivariate Gaussian case) and can be also applied to count linear regression.

Statistics Theory
Regression models of Pearson correlation coefficient

Abdisa G. Dufera,Tiantian Liu,Jin Xu

DOI: https://doi.org/10.1080/24754269.2023.2164970

2023-01-13

Statistical Theory and Related Fields

Abstract:We propose two simple regression models of Pearson correlation coefficient of two normal responses or binary responses to assess the effect of covariates of interest. Likelihood-based inference is established to estimate the regression coefficients, upon which bootstrap-based method is used to test the significance of covariates of interest. Simulation studies show the effectiveness of the method in terms of type-I error control, power performance in moderate sample size and robustness with respect to model mis-specification. We illustrate the application of the proposed method to some real data concerning health measurements.

Correlation Estimation System Minimization Compared to Least Squares Minimization in Simple Linear Regression

Iteratively Reweighted Least Squares Method for Estimating Polyserial and Polychoric Correlation Coefficients

On the minimax robustness against correlation and heteroscedasticity of ordinary least squares among generalized least squares estimators of regression

Novel Robust Least-Squares Estimator for Linear Dynamic Data Reconciliation

Predictive Regressions: Method of Least Autocorrelation

Statistical Inference for a Robust Measure of Multiple Correlation

Robust Estimating Equations and Bias Correction of Correlation Parameters for Longitudinal Data

Estimators of the correlation coefficient in the bivariate exponential distribution

Asymptotic Theory of Least Distances Estimate in Multivariate Linear Models

Robust Estimation of Polychoric Correlation

Optimal low-rank approximation to a correlation matrix

Partial Least Squares Methods: Partial Least Squares Correlation and Partial Least Square Regression

Kendall Correlation Coefficients for Portfolio Optimization

A compromise criterion for weighted least squares estimates

Least Squares Inference for Data with Network Dependency

Correlation Estimation in Hybrid Systems

With random regressors, least squares inference is robust to correlated errors with unknown correlation structure

Asymptotic confidence interval for R2 in multiple linear regression

Regression models of Pearson correlation coefficient