Discussion of "Vintage factor analysis with Varimax performs statistical inference"

Rungang Han,Anru R. Zhang

DOI: https://doi.org/10.48550/arXiv.2205.10151

2022-05-20

Abstract:We wholeheartedly congratulate Drs. Rohe and Zeng for their insightful paper \cite{rohe2020vintage} on vintage factor analysis with Varimax rotation. This note discusses the conditions to guarantee Varimax consistently recovers the subspace rotation.

Statistics Theory

What problem does this paper attempt to address?

Optimal vintage factor analysis with deflation varimax

Xin Bing,Dian Jin,Yuqian Zhang

2024-09-25

Abstract:Vintage factor analysis is one important type of factor analysis that aims to first find a low-dimensional representation of the original data, and then to seek a rotation such that the rotated low-dimensional representation is scientifically meaningful. The most widely used vintage factor analysis is the Principal Component Analysis (PCA) followed by the varimax rotation. Despite its popularity, little theoretical guarantee can be provided to date mainly because varimax rotation requires to solve a non-convex optimization over the set of orthogonal matrices. In this paper, we propose a deflation varimax procedure that solves each row of an orthogonal matrix sequentially. In addition to its net computational gain and flexibility, we are able to fully establish theoretical guarantees for the proposed procedure in a broader context. Adopting this new deflation varimax as the second step after PCA, we further analyze this two step procedure under a general class of factor models. Our results show that it estimates the factor loading matrix in the minimax optimal rate when the signal-to-noise-ratio (SNR) is moderate or large. In the low SNR regime, we offer possible improvement over using PCA and the deflation varimax when the additive noise under the factor model is structured. The modified procedure is shown to be minimax optimal in all SNR regimes. Our theory is valid for finite sample and allows the number of the latent factors to grow with the sample size as well as the ambient dimension to grow with, or even exceed, the sample size. Extensive simulation and real data analysis further corroborate our theoretical findings.

Machine Learning,Information Theory,Signal Processing
On varimax asymptotics in network models and spectral methods for dimensionality reduction

Joshua Cape

2024-03-09

Abstract:Varimax factor rotations, while popular among practitioners in psychology and statistics since being introduced by H. Kaiser, have historically been viewed with skepticism and suspicion by some theoreticians and mathematical statisticians. Now, work by K. Rohe and M. Zeng provides new, fundamental insight: varimax rotations provably perform statistical estimation in certain classes of latent variable models when paired with spectral-based matrix truncations for dimensionality reduction. We build on this newfound understanding of varimax rotations by developing further connections to network analysis and spectral methods rooted in entrywise matrix perturbation analysis. Concretely, this paper establishes the asymptotic multivariate normality of vectors in varimax-transformed Euclidean point clouds that represent low-dimensional node embeddings in certain latent space random graph models. We address related concepts including network sparsity, data denoising, and the role of matrix rank in latent variable parameterizations. Collectively, these findings, at the confluence of classical and contemporary multivariate analysis, reinforce methodology and inference procedures grounded in matrix factorization-based techniques. Numerical examples illustrate our findings and supplement our discussion.

Statistics Theory,Methodology
Strongly Hierarchical Factorization Machines and ANOVA Kernel Regression

Ruocheng Guo,Hamidreza Alvari,Paulo Shakarian

DOI: https://doi.org/10.48550/arXiv.1712.09133

2018-01-06

Abstract:High-order parametric models that include terms for feature interactions are applied to various data mining tasks, where ground truth depends on interactions of features. However, with sparse data, the high- dimensional parameters for feature interactions often face three issues: expensive computation, difficulty in parameter estimation and lack of structure. Previous work has proposed approaches which can partially re- solve the three issues. In particular, models with factorized parameters (e.g. Factorization Machines) and sparse learning algorithms (e.g. FTRL-Proximal) can tackle the first two issues but fail to address the third. Regarding to unstructured parameters, constraints or complicated regularization terms are applied such that hierarchical structures can be imposed. However, these methods make the optimization problem more challenging. In this work, we propose Strongly Hierarchical Factorization Machines and ANOVA kernel regression where all the three issues can be addressed without making the optimization problem more difficult. Experimental results show the proposed models significantly outperform the state-of-the-art in two data mining tasks: cold-start user response time prediction and stock volatility prediction.

Machine Learning
EIGENVALUE DISTRIBUTIONS OF VARIANCE COMPONENTS ESTIMATORS IN HIGH-DIMENSIONAL RANDOM EFFECTS MODELS

Fan Zhou,Iain M Johnstone

DOI: https://doi.org/10.1214/18-AOS1767

Abstract:We study the spectra of MANOVA estimators for variance component covariance matrices in multivariate random effects models. When the dimensionality of the observations is large and comparable to the number of realizations of each random effect, we show that the empirical spectra of such estimators are well-approximated by deterministic laws. The Stieltjes transforms of these laws are characterized by systems of fixed-point equations, which are numerically solvable by a simple iterative procedure. Our proof uses operator-valued free probability theory, and we establish a general asymptotic freeness result for families of rectangular orthogonally-invariant random matrices, which is of independent interest. Our work is motivated in part by the estimation of components of covariance between multiple phenotypic traits in quantitative genetics, and we specialize our results to common experimental designs that arise in this application.
Geometric vector orthogonal rotation method in multiple-factor analysis

Shigeo Kashiwagi

DOI: https://doi.org/10.1007/bf02289541

1965-12-01

Psychometrika

Abstract:An objective method for the orthogonal rotation of factors which gives results closer to the graphic method is proposed. First, the fact that the varimax method does not always satisfy simple-structure criteria, e.g., the positive manifold and the level contributions of all factors, is pointed out. Next, the principles of our method which are based on “geometric vector” are discussed, and the computational procedures for this method are explained using Harman and Holzinger's eight physical variables. Finally, six numerical examples by our method are presented, and it is shown that they are very close to the factors obtained from empirical studies both in values and in signs.

psychology, mathematical,mathematics, interdisciplinary applications,social sciences, mathematical methods
On limiting spectral distribution of large sample covariance matrices by VARMA(p,q)

Cheng Wang,Baisuo Jin,Baiqi Miao

DOI: https://doi.org/10.1111/j.1467-9892.2010.00712.x

2011-01-01

Journal of Time Series Analysis

Abstract:We studied the limiting spectral distribution of large-dimensional sample covariance matrices of a stationary and invertible VARMA(p,q) model. Relationship of the power spectral density and limiting spectral distribution of large population dimensional covariance matrices of ARMA(p,q) is established. The equation about Stieltjes transform of large-dimensional sample covariance matrices is also derived. As applications, the classical M-P law, VAR(1) and VMA(1) can be regarded as special examples.
Identity Tests for High Dimensional Data Using Rmt

Cheng Wang,Jing Yang,Baiqi Miao,Longbing Cao

DOI: https://doi.org/10.1016/j.jmva.2013.03.015

2013-01-01

Abstract:In this work, we redefined two important statistics, the CLRT test [Z. Bai, D. Jiang, J. Yao, S. Zheng, Corrections to LRT on large-dimensional covariance matrix by RMT, The Annals of Statistics 37 (6B) (2009) 3822–3840] and the LW test [O. Ledoit, M. Wolf, Some hypothesis tests for the covariance matrix when the dimension is large compared to the sample size, The Annals of Statistics (2002) 1081–1102] on identity tests for high dimensional data using random matrix theories. Compared with existing CLRT and LW tests, the new tests can accommodate data which has unknown means and non-Gaussian distributions. Simulations demonstrate that the new tests have good properties in terms of size and power. What is more, even for Gaussian data, our new tests perform favorably in comparison to existing tests. Finally, we find the CLRT is more sensitive to eigenvalues less than 1 while the LW test has more advantages in relation to detecting eigenvalues larger than 1.
A note on variational Bayesian factor analysis.

Jian-hua Zhao,Philip L H Yu

DOI: https://doi.org/10.1016/j.neunet.2008.11.002

IF: 7.8

2009-01-01

Neural Networks

Abstract:Existing works on variational bayesian (VB) treatment for factor analysis (FA) model such as [Ghahramani, Z., & Beal, M. (2000). Variational inference for Bayesian mixture of factor analysers. In Advances in neural information proceeding systems. Cambridge, MA: MIT Press; Nielsen, F. B. (2004). Variational approach to factor analysis and related models. Master's thesis, The Institute of Informatics and Mathematical Modelling, Technical University of Denmark.] are found theoretically and empirically to suffer two problems: (1) penalize the model more heavily than BIC and (2) perform unsatisfactorily in low noise cases as redundant factors can not be effectively suppressed. A novel VB treatment is proposed in this paper to resolve the two problems and a simulation study is conducted to testify its improved performance over existing treatments.
High-dimensional Factor Model and Its Applications to Statistical Machine Learning

Zhao Chen,Jianqing Fan,Christina Dan Wang

DOI: https://doi.org/10.1360/ssm-2020-0041

2020-01-01

Scientia Sinica Mathematica

Abstract:This paper reviews the recent developments on factor model and its applications to statistical machine learning. The factor model reduces the dimensionality of variables, and provides a low-rank plus sparse structure for the high-dimensional covariance matrices. Therefore, it attracts much attention in high-dimensional data analysis, and has been widely applied in many fields of sciences, engineering, humanities and social sciences, including economics, finance, genomics, neuroscience, machine learning, and so on. We elaborate how to use principal component analysis method to extract latent factors, estimate their associated factor loadings, idiosyncratic components, and their associated covariance matrices. These methods have been proven to effectively cope with the challenges of big data, such as high dimensionality, strong dependence, heavy-tailed variables, and heterogeneity. In addition, we also focus on the role of the factor model in dealing with high-dimensional statistical learning problems such as covariance matrix estimation, model selection, multiple testing, and prediction. Finally, we illustrate the innate relationships between factor models and modern machine learning problems through several applications, including network analysis, matrix completion, ranking, and mixture models.
Estimation of Variances and Covariances for High-Dimensional Data: A Selective Review

Tiejun Tong,Cheng Wang,Yuedong Wang

DOI: https://doi.org/10.1002/wics.1308

2014-01-01

Abstract:Estimation of variances and covariances is required for many statistical methods such as t-test, principal component analysis and linear discriminant analysis. High-dimensional data such as gene expression microarray data and financial data pose challenges to traditional statistical and computational methods. In this paper, we review some recent developments in the estimation of variances, covariance matrix, and precision matrix, with emphasis on the applications to microarray data analysis. (C) 2014 Wiley Periodicals, Inc.
Identifying Observed Factors in Approximate Factor Models: Estimation and Hypothesis Testing

Chen Liang

2012-01-01

Abstract:Despite their popularities in recent years, factor models have long been criticized for the lack of identification. Even when a large number of variables are available, the factors can only be consistently estimated up to a rotation. In this paper, we try to identify the underlying factors by associating them to a set of observed variables, and thus give interpretations to the orthogonal factors estimated by the method of Principal Components. We first propose a estimation procedure to select a set of observed variables, and then test the hypothesis that true factors are exact linear combinations of the selected variables. Our estimation method is shown to able to correctly identity the true observed factor even in the presence of mild measurement errors, and our test statistics are shown to be more general than those of Bai and Ng (2006). The applicability of our methods in finite samples and the advantages of our tests are confirmed by simulations. Our methods are also applied to the returns of portfolios to identify the underlying risk factors.
Static and Dynamic Models for Multivariate Distribution Forecasts: Proper Scoring Rule Tests of Factor-Quantile vs. Multivariate GARCH Models

Carol Alexander,Yang Han

DOI: https://doi.org/10.48550/arXiv.2004.14108

2022-03-10

Abstract:A plethora of static and dynamic models exist to forecast Value-at-Risk and other quantile-related metrics used in financial risk management. Industry practice tends to favour simpler, static models such as historical simulation or its variants whereas most academic research centres on dynamic models in the GARCH family. While numerous studies examine the accuracy of multivariate models for forecasting risk metrics, there is little research on accurately predicting the entire multivariate distribution. Yet this is an essential element of asset pricing or portfolio optimization problems having non-analytic solutions. We approach this highly complex problem using a variety of proper multivariate scoring rules to evaluate over 100,000 forecasts of eight-dimensional multivariate distributions: of exchange rates, interest rates and commodity futures. This way we test the performance of static models, viz. empirical distribution functions and a new factor-quantile model, with commonly used dynamic models in the asymmetric multivariate GARCH class.

Methodology
Analysis of Ordinal Variables Using Rank-Based Polychoric Correlation

Hao Luo,Johan Lyhagen,Fan Yang‐Wallentin

2011-01-01

Abstract:This thesis, which consists of five papers, is concerned with various aspects of confirmatory factor analysis (CFA) of ordinal variables and the generation of non-normal data. The first paper studies the performances of different estimation methods used in CFA when ordinal data are encountered. To take ordinality into account the four estimation methods, i.e., maximum likelihood (ML), unweighted least squares, diagonally weighted least squares, and weighted least squares (WLS), are used in combination with polychoric correlations. The effect of model sizes and number of categories on the parameter estimates, their standard errors, and the common chi-square measure of fit when the models are both correct and misspecified are examined. The second paper focuses on the appropriate estimator of the polychoric correlation when fitting a CFA model. A non-parametric polychoric correlation coefficient based on the discrete version of Spearman's rank correlation is proposed to contend with the situation of non-normal underlying distributions. The simulation study shows the benefits of using the non-parametric polychoric correlation under conditions of non-normality. The third paper raises the issue of simultaneous factor analysis. We study the effect of pooling multi-group data on the estimation of factor loadings. Given the same factor loadings but different factor means and correlations, we investigate how much information is lost by pooling the groups together and only estimating the combined data set using the WLS method. The parameter estimates and their standard errors are compared with results obtained by multi-group analysis using ML. The fourth paper uses a Monte Carlo simulation to assess the reliability of the Fleishman's power method under various conditions of skewness, kurtosis, and sample size. Based on the generated non-normal samples, the power of D'Agostino's (1986) normality test is studied. The fifth paper extends the evaluation of algorithms to the generation of multivariate non-normal data. Apart from the requirement of generating reliable skewness and kurtosis, the generated data also need to possess the desired correlation matrices. Four algorithms are investigated in terms of simplicity, generality, and reliability of the technique.
Dr Zhou Replies

Xiao-Hua Zhou,Nancy Obuchowski,Donna McClish

DOI: https://doi.org/10.1016/s1076-6332(03)00251-4

IF: 5.482

2003-01-01

Academic Radiology

Abstract:It seems that our remarks ( 1 Zhou X.H. Obuchowski N.A. McClish D.K. Statistical Methods in Diagnostic Medicine. Wiley-Interscience, New York, NY2002 Crossref Google Scholar ) concerning Beiden, Wagner, and Campbell’s (BWC) variance component analysis of multireader ROC (receiver operating characteristic) data ( 2 Beiden S.V. Wagner R.F. Campbell G. Components-of-variance models and multiple-bootstrap experiments. Acad Radiol. 2000; 7: 341-349 Abstract Full Text PDF PubMed Scopus (109) Google Scholar ) have offended some ( 3 Wagner R.F. Beiden S.V. Campbell G. An approach to multiple-reader, multiple-case (MRMC) ROC Analysis controversial - or subtle?. Acad Radiol. 2003; Google Scholar ); we apologize for this. We corresponded with these authors before writing this section of our book ( 1 Zhou X.H. Obuchowski N.A. McClish D.K. Statistical Methods in Diagnostic Medicine. Wiley-Interscience, New York, NY2002 Crossref Google Scholar ) to discuss our concerns about their methodology. We had hoped that their response would alleviate our concerns; it did not, so we proceeded with voicing our concerns in the book and meeting the editor’s fast-approaching publication deadline. It is unfortunate to both parties that these issues were not resolved before the book’s publication. We recognize the scientific merit of open-forum two-way debates of such issues, and we again apologize to BWC and users of their methodology (existing and future).
High-dimensional VARs with common factors

Ke Miao,Peter C.B. Phillips,Liangjun Su

DOI: https://doi.org/10.1016/j.jeconom.2022.02.002

IF: 3.363

2022-03-01

Journal of Econometrics

Abstract:This paper studies high-dimensional vector autoregressions (VARs) augmented with common factors that allow for strong cross-sectional dependence. Models of this type provide a convenient mechanism for accommodating the interconnectedness and temporal co-variability that are often present in large dimensional systems. We propose an ℓ1-nuclear-norm regularized estimator and derive the non-asymptotic upper bounds for the estimation errors as well as large sample asymptotics for the estimates. A singular value thresholding procedure is used to determine the correct number of factors with probability approaching one. Both the LASSO estimator and the conservative LASSO estimator are employed to improve estimation precision. The conservative LASSO estimates of the non-zero coefficients are shown to be asymptotically equivalent to the oracle least squares estimates. Simulations demonstrate that our estimators perform reasonably well in finite samples given the complex high-dimensional nature of the model. In an empirical illustration we apply the methodology to explore dynamic connectedness in the volatilities of financial asset prices and the transmission of 'investor fear'. The findings reveal that a large proportion of connectedness is due to the common factors. Conditional on the presence of these common factors, the results still document remarkable connectedness due to the interactions between the individual variables, thereby supporting a common factor augmented VAR specification.

economics,social sciences, mathematical methods,mathematics, interdisciplinary applications
Permutation methods for factor analysis and PCA

Edgar Dobriban

DOI: https://doi.org/10.48550/arXiv.1710.00479

2019-09-13

Abstract:Researchers often have datasets measuring features $x_{ij}$ of samples, such as test scores of students. In factor analysis and PCA, these features are thought to be influenced by unobserved factors, such as skills. Can we determine how many components affect the data? This is an important problem, because it has a large impact on all downstream data analysis. Consequently, many approaches have been developed to address it. Parallel Analysis is a popular permutation method. It works by randomly scrambling each feature of the data. It selects components if their singular values are larger than those of the permuted data. Despite widespread use in leading textbooks and scientific publications, as well as empirical evidence for its accuracy, it currently has no theoretical justification. In this paper, we show that the parallel analysis permutation method consistently selects the large components in certain high-dimensional factor models. However, it does not select the smaller components. The intuition is that permutations keep the noise invariant, while "destroying" the low-rank signal. This provides justification for permutation methods in PCA and factor models under some conditions. Our work uncovers drawbacks of permutation methods, and paves the way to improvements.

Statistics Theory,Methodology
FactorVAE: A Probabilistic Dynamic Factor Model Based on Variational Autoencoder for Predicting Cross-Sectional Stock Returns

Yitong Duan,Lei Wang,Qizhong Zhang,Jian Li

DOI: https://doi.org/10.1609/aaai.v36i4.20369

2022-06-28

Proceedings of the AAAI Conference on Artificial Intelligence

Abstract:As an asset pricing model in economics and finance, factor model has been widely used in quantitative investment. Towards building more effective factor models, recent years have witnessed the paradigm shift from linear models to more flexible nonlinear data-driven machine learning models. However, due to low signal-to-noise ratio of the financial data, it is quite challenging to learn effective factor models. In this paper, we propose a novel factor model, FactorVAE, as a probabilistic model with inherent randomness for noise modeling. Essentially, our model integrates the dynamic factor model (DFM) with the variational autoencoder (VAE) in machine learning, and we propose a prior-posterior learning method based on VAE, which can effectively guide the learning of model by approximating an optimal posterior factor model with future information. Particularly, considering that risk modeling is important for the noisy stock data, FactorVAE can estimate the variances from the distribution over the latent space of VAE, in addition to predicting returns. The experiments on the real stock market data demonstrate the effectiveness of FactorVAE, which outperforms various baseline methods.
An algorithm for portfolio's value at risk based on principal factor analysis

Honggang Xue,Chengxian Xu,Chunping Hu

DOI: https://doi.org/10.1007/11496199_41

2005-01-01

Abstract:In this paper, we propose principle factor analysis method to reduce the dimensions of a high dimensional random vector in calculating portfolio's Value at Risk. The theoretical foundation, algorithm and numerical example of the method are given. This method outperforms the principle component analysis method. Especially, the advantages of the method are marked, while the factors F's multicollinearity is serious.
Looking back: Selected contributions by C. R. Rao to multivariate analysis

Dianna Smith

DOI: https://doi.org/10.1111/sjos.12749

2024-08-28

Scandinavian Journal of Statistics

Abstract:Statistician C. R. Rao made many contributions to multivariate analysis over the span of his career. Some of his earliest contributions continue to be used and built upon almost 80 years later, while his more recent contributions spur new avenues of research. The present article discusses these contributions, how they helped shape multivariate analysis as we see it today, and what we may learn from reviewing his works. Topics include his extension of linear discriminant analysis, Rao's perimeter test, Rao's U statistic, his asymptotic expansion of Wilks' Λ statistic, canonical factor analysis, functional principal component analysis, redundancy analysis, canonical coordinates, and correspondence analysis. The examination of his works shows that interdisciplinary collaboration and the utilization of real datasets were crucial in almost all of Rao's impactful contributions.

statistics & probability
A Direct Estimation of High Dimensional Stationary Vector Autoregressions

Fang Han,Huanran Lu,Han Liu

DOI: https://doi.org/10.48550/arXiv.1307.0293

2014-10-29

Abstract:The vector autoregressive (VAR) model is a powerful tool in modeling complex time series and has been exploited in many fields. However, fitting high dimensional VAR model poses some unique challenges: On one hand, the dimensionality, caused by modeling a large number of time series and higher order autoregressive processes, is usually much higher than the time series length; On the other hand, the temporal dependence structure in the VAR model gives rise to extra theoretical challenges. In high dimensions, one popular approach is to assume the transition matrix is sparse and fit the VAR model using the "least squares" method with a lasso-type penalty. In this manuscript, we propose an alternative way in estimating the VAR model. The main idea is, via exploiting the temporal dependence structure, to formulate the estimating problem into a linear program. There is instant advantage for the proposed approach over the lasso-type estimators: The estimation equation can be decomposed into multiple sub-equations and accordingly can be efficiently solved in a parallel fashion. In addition, our method brings new theoretical insights into the VAR model analysis. So far the theoretical results developed in high dimensions (e.g., Song and Bickel (2011) and Kock and Callot (2012)) mainly pose assumptions on the design matrix of the formulated regression problems. Such conditions are indirect about the transition matrices and not transparent. In contrast, our results show that the operator norm of the transition matrices plays an important role in estimation accuracy. We provide explicit rates of convergence for both estimation and prediction. In addition, we provide thorough experiments on both synthetic and real-world equity data to show that there are empirical advantages of our method over the lasso-type estimators in both parameter estimation and forecasting.

Machine Learning

Discussion of "Vintage factor analysis with Varimax performs statistical inference"

Optimal vintage factor analysis with deflation varimax

On varimax asymptotics in network models and spectral methods for dimensionality reduction

Strongly Hierarchical Factorization Machines and ANOVA Kernel Regression

EIGENVALUE DISTRIBUTIONS OF VARIANCE COMPONENTS ESTIMATORS IN HIGH-DIMENSIONAL RANDOM EFFECTS MODELS

Geometric vector orthogonal rotation method in multiple-factor analysis

On limiting spectral distribution of large sample covariance matrices by VARMA(p,q)

Identity Tests for High Dimensional Data Using Rmt

A note on variational Bayesian factor analysis.

High-dimensional Factor Model and Its Applications to Statistical Machine Learning

Estimation of Variances and Covariances for High-Dimensional Data: A Selective Review

Identifying Observed Factors in Approximate Factor Models: Estimation and Hypothesis Testing

Static and Dynamic Models for Multivariate Distribution Forecasts: Proper Scoring Rule Tests of Factor-Quantile vs. Multivariate GARCH Models

Analysis of Ordinal Variables Using Rank-Based Polychoric Correlation

Dr Zhou Replies

High-dimensional VARs with common factors

Permutation methods for factor analysis and PCA

FactorVAE: A Probabilistic Dynamic Factor Model Based on Variational Autoencoder for Predicting Cross-Sectional Stock Returns

An algorithm for portfolio's value at risk based on principal factor analysis

Looking back: Selected contributions by C. R. Rao to multivariate analysis

A Direct Estimation of High Dimensional Stationary Vector Autoregressions