Chapter 6: Ordinary Least Squares

DOI: https://doi.org/10.1137/1.9781611977905.ch6

2024-03-27

Theory of Probability and Its Applications

Abstract:Other Titles in Applied Mathematics, Page 51-61. Excerpt Notation: In this chapter, we use the traditional notation for regression in financial economics. The parameter vector, instead of θ, is denoted by b. Its true value θ* is denoted by β.

statistics & probability

What problem does this paper attempt to address?

Chapter 7: Generalized Least Squares (GLS)

DOI: https://doi.org/10.1137/1.9781611977905.ch7

2024-03-27

Theory of Probability and Its Applications

Abstract:Other Titles in Applied Mathematics, Page 63-65. Excerpt We consider the linear regression problem We are going to reuse Assumption 1 from Chapter 6, that is, But now we suppose that the errors u(ω) are autocorrelated, that is, V is no longer diagonal. In this case, the Gauss-Markov theorem does not hold anymore. A simple transformation of variables allows us to return to the case where the Gauss-Markov theorem applies.

statistics & probability
A Least Squares Estimator for Lévy-driven Moving Averages Based on Discrete Time Observations

Shibin Zhang,Zhengyan Lin,Xinsheng Zhang

DOI: https://doi.org/10.1080/03610926.2012.763093

2013-01-01

Abstract:This article is concerned with a least squares estimator (LSE) of the kernel function parameter theta for a Levy-driven moving average of the form X(t) = integral(t)(-infinity) K(theta(t - s)) dL(s), where L = {L(t), t is an element of R} is a Levy process without the Brownian motion part, K is a kernel function and theta > 0 is a parameter. Let h be the time span between two consecutive observations and let n be the size of sample. As h -> 0 and nh -> infinity, consistency and asymptotic normality of the LSE are studied. The small-sample performance of the LSE is evaluated by means of a simulation experiment. Finally, two real-data applications show that the Levy-driven moving average gives a good approximation to the autocorrelation of the process.
Model-free Study of Ordinary Least Squares Linear Regression

Arun K. Kuchibhotla,Lawrence D. Brown,Andreas Buja

DOI: https://doi.org/10.48550/arXiv.1809.10538

2018-09-27

Abstract:Ordinary least squares (OLS) linear regression is one of the most basic statistical techniques for data analysis. In the main stream literature and the statistical education, the study of linear regression is typically restricted to the case where the covariates are fixed, errors are mean zero Gaussians with variance independent of the (fixed) covariates. Even though OLS has been studied under misspecification from as early as the 1960's, the implications have not yet caught up with the main stream literature and applied sciences. The present article is an attempt at a unified viewpoint that makes the various implications of misspecification stand out.

Statistics Theory
Understanding Regression Analysis: An Introductory Guide

Larry D. Schroeder,David L. Sjoquist,Paula E. Stephan

DOI: https://doi.org/10.4135/9781506361628

2017-01-01

Abstract:Understanding Regression Analysis: An Introductory Guide presents the fundamentals of regression analysis, from its meaning to uses, in a concise, easy-to-read, and non-technical style. It illustrates how regression coefficients are estimated, interpreted, and used in a variety of settings within the social sciences, business, law, and public policy. Packed with applied examples and using few equations, the book walks readers through elementary material using a verbal, intuitive interpretation of regression coefficients, associated statistics, and hypothesis tests. The Second Edition features updated examples and new references to modern software output.
Financial Econometrics, Mathematics and Statistics

Cheng-Few Lee,Hong-Yi Chen,John Lee

DOI: https://doi.org/10.1007/978-1-4939-9429-8

2019-01-01

Abstract:This rigorous textbook introduces graduate students to the principles of econometrics and statistics with a focus on methods and applications in financial research. Financial Econometrics, Mathematics, and Statistics illustrates tools and methods important for both finance and accounting that assist with asset pricing, corporate finance, options and futures, and conducting financial accounting research. Divided into four parts, the text offers insight into the following models and topics, among others: • Multiple linear regression • Time-series analysis • Option pricing models • Risk management • Heteroskedasticity • Itô’s Calculus • Spurious regression • Errors-in-variable Written by leading academics in the quantitative finance field, this book allows readers to implement the principles behind financial econometrics and statistics through real-world applications and problem sets. It will appeal to a less-served market of advanced students and scholars in finance, economics, accounting, and statistics.
A rigorous introduction to linear models

Jun Lu

2024-02-04

Abstract:This book is meant to provide an introduction to linear models and the theories behind them. Our goal is to give a rigorous introduction to the readers with prior exposure to ordinary least squares. In machine learning, the output is usually a nonlinear function of the input. Deep learning even aims to find a nonlinear dependence with many layers, which require a large amount of computation. However, most of these algorithms build upon simple linear models. We then describe linear models from different perspectives and find the properties and theories behind the models. The linear model is the main technique in regression problems, and the primary tool for it is the least squares approximation, which minimizes a sum of squared errors. This is a natural choice when we're interested in finding the regression function which minimizes the corresponding expected squared error. This book is primarily a summary of purpose, significance of important theories behind linear models, e.g., distribution theory and the minimum variance estimator. We first describe ordinary least squares from three different points of view, upon which we disturb the model with random noise and Gaussian noise. Through Gaussian noise, the model gives rise to the likelihood so that we introduce a maximum likelihood estimator. It also develops some distribution theories via this Gaussian disturbance. The distribution theory of least squares will help us answer various questions and introduce related applications. We then prove least squares is the best unbiased linear model in the sense of mean squared error, and most importantly, it actually approaches the theoretical limit. We end up with linear models with the Bayesian approach and beyond.

Machine Learning
Least Squares Learning? Evidence from the Laboratory

Te Bao,Yun Dai,John Duffy

DOI: https://doi.org/10.1016/j.jedc.2024.104980

IF: 1.62

2024-11-10

Journal of Economic Dynamics and Control

Abstract:We report on an experiment testing the empirical relevance of least squares (LS) learning, a common way of modelling how individuals learn a rational expectations equilibrium (REE). Subjects are endowed with the correct perceived law of motion (PLM) for a price level variable they are seeking to forecast, but do not know the true parameterization of that PLM. Instead, they must choose and can adjust the parameters of this PLM over 50 periods. Consistent with the E-stability of the REE in the model studied, 97.8% of subjects achieve weak convergence to the REE in terms of their price level predictions. However, the number of participants that can be characterized as least squares learners via the adjustments they make to the parameterization of the PLM over time depends on properties of the data generating process of the dependent and independent variables. Participants learn the REE faster, and behave more like least squares learners when there is greater variance in the independent variable of the model. We consider several alternatives to least squares learning and find evidence that many subjects employ a simple satisficing approach.

economics
Weighted asymmetric least squares regression with fixed-effects

Amadou Barry,Karim Oualkacha,Arthur Charpentier

DOI: https://doi.org/10.48550/arXiv.2108.04737

2021-08-10

Econometrics

Abstract:The fixed-effects model estimates the regressor effects on the mean of the response, which is inadequate to summarize the variable relationships in the presence of heteroscedasticity. In this paper, we adapt the asymmetric least squares (expectile) regression to the fixed-effects model and propose a new model: expectile regression with fixed-effects $(\ERFE).$ The $\ERFE$ model applies the within transformation strategy to concentrate out the incidental parameter and estimates the regressor effects on the expectiles of the response distribution. The $\ERFE$ model captures the data heteroscedasticity and eliminates any bias resulting from the correlation between the regressors and the omitted factors. We derive the asymptotic properties of the $\ERFE$ estimators and suggest robust estimators of its covariance matrix. Our simulations show that the $\ERFE$ estimator is unbiased and outperforms its competitors. Our real data analysis shows its ability to capture data heteroscedasticity (see our R package, \url{github.com/AmBarry/erfe}).
Financial Economics and Econometrics

Nikiforos T. Laopodis

DOI: https://doi.org/10.4324/9781003205005

2021-11-05

Abstract:Financial Economics and Econometrics provides an overview of the core topics in theoretical and empirical finance, with an emphasis on applications and interpreting results. Structured in five parts, the book covers financial data and univariate models; asset returns; interest rates, yields and spreads; volatility and correlation; and corporate finance and policy. Each chapter begins with a theory in financial economics, followed by econometric methodologies which have been used to explore the theory. Next, the chapter presents empirical evidence and discusses seminal papers on the topic. Boxes offer insights on how an idea can be applied to other disciplines such as management, marketing and medicine, showing the relevance of the material beyond finance. Readers are supported with plenty of worked examples and intuitive explanations throughout the book, while key takeaways, ‘test your knowledge’ and ‘test your intuition’ features at the end of each chapter also aid student learning. Digital supplements including PowerPoint slides, computer codes supplements, an Instructor’s Manual and Solutions Manual are available for instructors. This textbook is suitable for upper-level undergraduate and graduate courses on financial economics, financial econometrics, empirical finance and related quantitative areas.
Bivariate least squares linear regression: towards a unified analytic formalism

R. Caimmi

DOI: https://doi.org/10.1016/j.newast.2011.01.001

2011-03-03

Abstract:Concerning bivariate least squares linear regression, the classical approach pursued for functional models in earlier attempts is reviewed using a new formalism in terms of deviation (matrix) traces. Within the framework of classical error models, the dependent variable relates to the independent variable according to the usual additive model. Linear models of regression lines are considered in the general case of correlated errors in X and in Y for heteroscedastic data. The special case of (C) generalized orthogonal regression is considered in detail together with well known subcases. In the limit of homoscedastic data, the results determined for functional models are compared with their counterparts related to extreme structural models. While regression line slope and intercept estimators for functional and structural models necessarily coincide, the contrary holds for related variance estimators even if the residuals obey a Gaussian distribution, with a single exception. An example of astronomical application is considered, concerning the [O/H]-[Fe/H] empirical relations deduced from five samples related to different stars and/or different methods of oxygen abundance determination. For selected samples and assigned methods, different regression models yield consistent results within the errors for both heteroscedastic and homoscedastic data. Conversely, samples related to different methods produce discrepant results, due to the presence of (still undetected) systematic errors, which implies no definitive statement can be made at present. A comparison is also made between different expressions of regression line slope and intercept variance estimators, where fractional discrepancies are found to be not exceeding a few percent, which grows up to about 20% in presence of large dispersion data.

Instrumentation and Methods for Astrophysics,Statistics Theory
Investment Volatility: A Critique of Standard Beta Estimation and a Simple Way Forward

Chris Tofallis

DOI: https://doi.org/10.48550/arXiv.1109.4422

2011-09-21

Abstract:Beta is a widely used quantity in investment analysis. We review the common interpretations that are applied to beta in finance and show that the standard method of estimation - least squares regression - is inconsistent with these interpretations. We present the case for an alternative beta estimator which is more appropriate, as well as being easier to understand and to calculate. Unlike regression, the line fit we propose treats both variables in the same way. Remarkably, it provides a slope that is precisely the ratio of the volatility of the investment's rate of return to the volatility of the market index rate of return (or the equivalent excess rates of returns). Hence, this line fitting method gives an alternative beta, which corresponds exactly to the relative volatility of an investment - which is one of the usual interpretations attached to beta.

Portfolio Management,Risk Management
THE ASYMPTOTICALLY OPTIMAL EMPIRICAL BAYES ESTIMATION IN MULTIPLE LINEAR REGRESSION MODEL

Zhang Shunpu,Wei Laisheng

DOI: https://doi.org/10.1007/bf02663774

1994-01-01

Abstract:Empirical Bayes estimation of the parameter vector θ=(β’,σ2)’ in a multiple linear regression modelY=Xβ+ε is considered, where β is the vector of regression coefficient, ε∽N(0,σI with σ2 unknown. In this paper, we construct the EB estimators of θ by using the kernel estimation of multivariate density function and its partial derivatives. Under some moment conditions on prior distribution we obtain their asymptotic optimality.
Variations on least squares

Erik Talvila

DOI: https://doi.org/10.48550/arXiv.2011.04836

2020-11-10

Abstract:Three methods of least squares are examined for fitting a line to points in the plane. Two well known methods are to minimize sums of squares of vertical or horizontal distances to the line. Less known is to minimize sums of squares of distances to the line. Concise proofs are given for each method using a combination of the first derivative test for functions of two variables and completing the square. The three methods are compared and the distances to the line method appears to be favourable in most circumstances. They generally draw different regression lines. The method of vertical displacements typically gives a slope of too small magnitude while the method of horizontal displacements typically gives a slope of too large magnitude. An inequality involving the three slopes is proved. Rotating all the data points in the same way with these two methods does not result in the regression line being rotated the same way. However, the distance to the line method is invariant under rotations.

Classical Analysis and ODEs
Rational Expectations Equilibrium

J. Stachurski,T. Sargent

Abstract:This lecture introduces the concept of rational expectations equilibrium. To illustrate it, we describe a linear quadratic version of a famous and important model due to Lucas and Prescott [4]. This 1971 paper is one of a small number of research articles that kicked off the rational expectations revolution. We follow Lucas and Prescott by employing a setting that is readily “Bellmanized” (i.e., capable of being formulated in terms of dynamic programming problems). Because we use linear quadratic setups for demand and costs, we can adapt the LQ programming techniques described in this lecture. We will learn about how a representative agent’s problem differs from a planner’s, and how a planning problem can be used to compute rational expectations quantities. We will also learn about how a rational expectations equilibrium can be characterized as a fixed point of a mapping from a perceived law of motion to an actual law of motion. Equality between a perceived and an actual law of motion for endogenous market-wide objects captures in a nutshell what the rational expectations equilibrium concept is all about. Finally, we will learn about the important “Big K, little k” trick, a modeling device widely used in macroeconomics. Except that for us

Economics
Estimating Unknown Parameters and Disturbance Term in Uncertain Regression Models by the Principle of Least Squares

Han Wang,Yang Liu,Haiyan Shi

DOI: https://doi.org/10.3390/sym16091182

2024-09-10

Symmetry

Abstract:In the field of statistics, uncertain regression analysis occupies an important position. It can thoroughly analyze data sets contained in complex uncertainties, aiming to quantify and reveal the intricate relationships between variables. It is worth noting that the traditional least squares method only takes into account the reduction in the deviations between predictions and observations, and fails to fully consider the inherent characteristics of the correlation uncertainty distributions under the uncertain regression framework. In light of this, this paper constructs a statistical invariant with symmetric uncertainty distribution based on the observations and the disturbance term. It also proposes the least squares estimation of unknown parameters and disturbance term in the uncertain regression model based on the least squares principle and, combined with the mathematical properties of the normal uncertainty distribution, gives a numerical algorithm for solving specific estimates. Finally, in order to verify the effectiveness of the least squares estimation method proposed in this paper, we also design two numerical examples and an empirical study of forecasting of electrical power output.

multidisciplinary sciences
Standard Errors for Regression-Based Causal Effect Estimates in Economics Using Numerical Derivatives

Joseph V. Terza

DOI: https://doi.org/10.1007/s10614-024-10565-w

IF: 1.741

2024-03-27

Computational Economics

Abstract:The aim of nearly all empirical studies in economics is to provide scientific evidence that can be used to assess policy relevant cause-and-effect. In the context of the general potential outcomes framework, we review how a causal effect parameter can be rigorously but tractably specified, identified and estimated along with its asymptotic standard error. For cases in which the analytic and computational requirements for calculation of the ASE are challenging, we suggest the use of numerical derivatives (ND). We detail the specific type of ND software required for this purpose, and note that it is offered as a feature in most statistical packages. As an illustration, we analyze the causal effect of wife's high school graduation on family size using the Stata/Mata deriv command. Code for this example is supplied in an appendix.

economics,mathematics, interdisciplinary applications,management
Least Absolute Relative Error Estimation

Kani Chen,Shaojun Guo,Yuanyuan Lin,Zhiliang Ying

DOI: https://doi.org/10.1198/jasa.2010.tm09307

IF: 4.369

2010-01-01

Journal of the American Statistical Association

Abstract:Multiplicative regression model or accelerated failure time model, which becomes linear regression model after logarithmic transformation, is useful in analyzing data with positive responses, such as stock prices or life times, that are particularly common in economic/financial or biomedical studies. Least squares or least absolute deviation are among the most widely used criterions in statistical estimation for linear regression model. However, in many practical applications, especially in treating, for example, stock price data, the size of relative error, rather than that of error itself, is the central concern of the practitioners. This paper offers an alternative to the traditional estimation methods by considering minimizing the least absolute relative errors for multiplicative regression models. We prove consistency and asymptotic normality and provide an inference approach via random weighting. We also specify the error distribution, with which the proposed least absolute relative errors estimation is efficient. Supportive evidence is shown in simulation studies. Application is illustrated in an analysis of stock returns in Hong Kong Stock Exchange.
Using Johnson Schumacher Model for Parameter Estimation of Nonlinear Regression Model

maie kamel,maie kamel,Hanaa Hanaa Abd El-Redeem Salem,Mona Nazih Abdel Bary Nazih Abdel Bary

DOI: https://doi.org/10.21608/jsst.2022.169520.1497

2022-10-01

مجلة البحوث المالیة والتجاریة

Abstract:In this study, we aim to estimate parameters of nonlinear model by using ordinary least square. This paper used a real data set on exchange rate, inflation, exports, imports, investments, and budget deficit. The appropriate models of the data are cubic and Johnson Schumacher Model. Both Results of application data and Simulation study appear that the Johnson Schumacher nonlinear regression model is outstanding performance This paper develops the reliable alternative approach of parameter estimation based on the PSO algorithm in the model of nonlinear regression. When the PSO method is used for estimating of nonlinear regression model parameters. Estimating (Forecasting) of the exchange rate using Johnson Schumacher nonlinear model This study presents the following variables: (X1: inflation), (X2: exports), (X3: imports), (X4: investments), (X5: budget deficit). These represent the main variables that affect (Y: exchange rate). The study introduced two nonlinear models cubic and Johnson Schumacher nonlinear model. The Johnson Schumacher nonlinear model shows outstanding performance. The results of the simulation
Least quartic Regression Criterion with Application to Finance

Giuseppe arbia

DOI: https://doi.org/10.48550/arXiv.1403.4171

2014-03-18

Abstract:This article proposes a new method for the estimation of the parameters of a simple linear regression model which accounts for the role of co-moments in non-Gaussian distributions being based on the minimization of a quartic loss function. Although the proposed method is very general, we examine its application to finance. In fact, in this field the contribution of the co-moments in explaining the return-generating process is of paramount importance when evaluating the systematic risk of an asset within the framework of the Capital Asset Pricing Model (CAPM). The suggested new method contributes to this literature by showing that, in the presence of non-normality, the regression slope can be expressed as a function of the co-kurtosis between the returns of a risky asset and the market proxy. The paper provides an illustration of the method based on some empirical financial data referring to 40 industrial sector assets rates of the Italian stock market.

Statistical Finance,Applications
Double Descent: Understanding Linear Model Estimation of Nonidentifiable Parameters and a Model for Overfitting

Ronald Christensen

2024-08-24

Abstract:We consider ordinary least squares estimation and variations on least squares estimation such as penalized (regularized) least squares and spectral shrinkage estimates for problems with p > n and associated problems with prediction of new observations. After the introduction of Section 1, Section 2 examines a number of commonly used estimators for p > n. Section 3 introduces prediction with p > n. Section 4 introduces notational changes to facilitate discussion of overfitting and Section 5 illustrates the phenomenon of double descent. We conclude with some final comments.

Machine Learning,Statistics Theory

Chapter 6: Ordinary Least Squares

Chapter 7: Generalized Least Squares (GLS)

A Least Squares Estimator for Lévy-driven Moving Averages Based on Discrete Time Observations

Model-free Study of Ordinary Least Squares Linear Regression

Understanding Regression Analysis: An Introductory Guide

Financial Econometrics, Mathematics and Statistics

A rigorous introduction to linear models

Least Squares Learning? Evidence from the Laboratory

Weighted asymmetric least squares regression with fixed-effects

Financial Economics and Econometrics

Bivariate least squares linear regression: towards a unified analytic formalism

Investment Volatility: A Critique of Standard Beta Estimation and a Simple Way Forward

THE ASYMPTOTICALLY OPTIMAL EMPIRICAL BAYES ESTIMATION IN MULTIPLE LINEAR REGRESSION MODEL

Variations on least squares

Rational Expectations Equilibrium

Estimating Unknown Parameters and Disturbance Term in Uncertain Regression Models by the Principle of Least Squares

Standard Errors for Regression-Based Causal Effect Estimates in Economics Using Numerical Derivatives

Least Absolute Relative Error Estimation

Using Johnson Schumacher Model for Parameter Estimation of Nonlinear Regression Model

Least quartic Regression Criterion with Application to Finance

Double Descent: Understanding Linear Model Estimation of Nonidentifiable Parameters and a Model for Overfitting