Multivariate Data Modeling Methods

Uwe Krüger,Lei Xie

DOI: https://doi.org/10.1002/9780470517253.ch2

2012-01-01

Abstract:This chapter contains sections titled: Principal component analysis Partial least squares Maximum redundancy partial least squares Estimating the number of source signals Tutorial Session

What problem does this paper attempt to address?

Maximum Likelihood Estimators in a Two Step Model for PLS

Ying Li,Dietrich von Rosen

DOI: https://doi.org/10.1080/03610926.2011.607531

2012-01-01

Abstract:Univariate partial least squares regression (PLS1) is a method of modeling relationships between a response variable and explanatory variables, especially when the explanatory variables are almost collinear. The purpose is to predict a future response observation, although in many applications there is an interest to understand the contributions of each explanatory variable. It is an algorithmic approach. In this article, we are going to use the algorithm presented by Helland (1988). The population PLS predictor is linked to a linear model including a Krylov design matrix and a two-step estimation procedure. For the first step, the maximum likelihood approach is applied to a specific multivariate linear model, generating tools for evaluating the information in the explanatory variables. It is shown that explicit maximum likelihood estimators of the dispersion matrix can be obtained where the dispersion matrix, besides representing the variation in the error, also includes the Krylov structured design matrix describing the mean.
Applied Multivariate Statistical Analysis

D. W. Wichern,Richard A. Johnson

DOI: https://doi.org/10.2307/2347962

1983-06-01

Abstract:(NOTE: Each chapter begins with an Introduction, and concludes with Exercises and References.) I. GETTING STARTED. 1. Aspects of Multivariate Analysis. Applications of Multivariate Techniques. The Organization of Data. Data Displays and Pictorial Representations. Distance. Final Comments. 2. Matrix Algebra and Random Vectors. Some Basics of Matrix and Vector Algebra. Positive Definite Matrices. A Square-Root Matrix. Random Vectors and Matrices. Mean Vectors and Covariance Matrices. Matrix Inequalities and Maximization. Supplement 2A Vectors and Matrices: Basic Concepts. 3. Sample Geometry and Random Sampling. The Geometry of the Sample. Random Samples and the Expected Values of the Sample Mean and Covariance Matrix. Generalized Variance. Sample Mean, Covariance, and Correlation as Matrix Operations. Sample Values of Linear Combinations of Variables. 4. The Multivariate Normal Distribution. The Multivariate Normal Density and Its Properties. Sampling from a Multivariate Normal Distribution and Maximum Likelihood Estimation. The Sampling Distribution of 'X and S. Large-Sample Behavior of 'X and S. Assessing the Assumption of Normality. Detecting Outliners and Data Cleaning. Transformations to Near Normality. II. INFERENCES ABOUT MULTIVARIATE MEANS AND LINEAR MODELS. 5. Inferences About a Mean Vector. The Plausibility of ...m0 as a Value for a Normal Population Mean. Hotelling's T 2 and Likelihood Ratio Tests. Confidence Regions and Simultaneous Comparisons of Component Means. Large Sample Inferences about a Population Mean Vector. Multivariate Quality Control Charts. Inferences about Mean Vectors When Some Observations Are Missing. Difficulties Due To Time Dependence in Multivariate Observations. Supplement 5A Simultaneous Confidence Intervals and Ellipses as Shadows of the p-Dimensional Ellipsoids. 6. Comparisons of Several Multivariate Means. Paired Comparisons and a Repeated Measures Design. Comparing Mean Vectors from Two Populations. Comparison of Several Multivariate Population Means (One-Way MANOVA). Simultaneous Confidence Intervals for Treatment Effects. Two-Way Multivariate Analysis of Variance. Profile Analysis. Repealed Measures, Designs, and Growth Curves. Perspectives and a Strategy for Analyzing Multivariate Models. 7. Multivariate Linear Regression Models. The Classical Linear Regression Model. Least Squares Estimation. Inferences About the Regression Model. Inferences from the Estimated Regression Function. Model Checking and Other Aspects of Regression. Multivariate Multiple Regression. The Concept of Linear Regression. Comparing the Two Formulations of the Regression Model. Multiple Regression Models with Time Dependant Errors. Supplement 7A The Distribution of the Likelihood Ratio for the Multivariate Regression Model. III. ANALYSIS OF A COVARIANCE STRUCTURE. 8. Principal Components. Population Principal Components. Summarizing Sample Variation by Principal Components. Graphing the Principal Components. Large-Sample Inferences. Monitoring Quality with Principal Components. Supplement 8A The Geometry of the Sample Principal Component Approximation. 9. Factor Analysis and Inference for Structured Covariance Matrices. The Orthogonal Factor Model. Methods of Estimation. Factor Rotation. Factor Scores. Perspectives and a Strategy for Factor Analysis. Structural Equation Models. Supplement 9A Some Computational Details for Maximum Likelihood Estimation. 10. Canonical Correlation Analysis Canonical Variates and Canonical Correlations. Interpreting the Population Canonical Variables. The Sample Canonical Variates and Sample Canonical Correlations. Additional Sample Descriptive Measures. Large Sample Inferences. IV. CLASSIFICATION AND GROUPING TECHNIQUES. 11. Discrimination and Classification. Separation and Classification for Two Populations. Classifications with Two Multivariate Normal Populations. Evaluating Classification Functions. Fisher's Discriminant Function...nSeparation of Populations. Classification with Several Populations. Fisher's Method for Discriminating among Several Populations. Final Comments. 12. Clustering, Distance Methods and Ordination. Similarity Measures. Hierarchical Clustering Methods. Nonhierarchical Clustering Methods. Multidimensional Scaling. Correspondence Analysis. Biplots for Viewing Sample Units and Variables. Procustes Analysis: A Method for Comparing Configurations. Appendix. Standard Normal Probabilities. Student's t-Distribution Percentage Points. ...c2 Distribution Percentage Points. F-Distribution Percentage Points. F-Distribution Percentage Points (...a = .10). F-Distribution Percentage Points (...a = .05). F-Distribution Percentage Points (...a = .01). Data Index. Subject Index.

Mathematics
Kernel Multivariate Analysis Framework for Supervised Subspace Learning: A Tutorial on Linear and Kernel Multivariate Methods

Jerónimo Arenas-García,Kaare Brandt Petersen,Gustavo Camps-Valls,Lars Kai Hansen

DOI: https://doi.org/10.1109/MSP.2013.2250591

2013-10-19

Abstract:Feature extraction and dimensionality reduction are important tasks in many fields of science dealing with signal processing and analysis. The relevance of these techniques is increasing as current sensory devices are developed with ever higher resolution, and problems involving multimodal data sources become more common. A plethora of feature extraction methods are available in the literature collectively grouped under the field of Multivariate Analysis (MVA). This paper provides a uniform treatment of several methods: Principal Component Analysis (PCA), Partial Least Squares (PLS), Canonical Correlation Analysis (CCA) and Orthonormalized PLS (OPLS), as well as their non-linear extensions derived by means of the theory of reproducing kernel Hilbert spaces. We also review their connections to other methods for classification and statistical dependence estimation, and introduce some recent developments to deal with the extreme cases of large-scale and low-sized problems. To illustrate the wide applicability of these methods in both classification and regression problems, we analyze their performance in a benchmark of publicly available data sets, and pay special attention to specific real applications involving audio processing for music genre prediction and hyperspectral satellite images for Earth and climate monitoring.

Machine Learning
A Multivariate Approach to Correlation Analysis Based on Random Matrix Theory

Markus Mller,Gerold Baier,Christian Rummel,Kaspar Schindler,Ulrich Stephani

DOI: https://doi.org/10.1002/9783527625192.ch16

2008-08-20

Abstract:This chapter contains sections titled: Introduction The Equal‐time Correlation Matrix Eigenvalues, Eigenvectors and Interrelations between Data Channels Random and Non‐random Level Repulsion RMT Measures: Motivation and Definition Application to a Test System Cluster Detection based on Eigenvectors Application to EEG Recordings Conclusions References Introduction The Equal‐time Correlation Matrix Eigenvalues, Eigenvectors and Interrelations between Data Channels Random and Non‐random Level Repulsion RMT Measures: Motivation and Definition Application to a Test System Cluster Detection based on Eigenvectors Application to EEG Recordings Conclusions References

neurosciences,clinical neurology
Dynamic Multivariate Functional Data Modeling Via Sparse Subspace Learning

Chen Zhang,Hao Yan,Seungho Lee,Jianjun Shi

DOI: https://doi.org/10.1080/00401706.2020.1800516

2021-01-01

Abstract:Multivariate functional data from a complex system are naturally high-dimensional and have a complex cross-correlation structure. The complexity of data structure can be observed as that (1) some functions are strongly correlated with similar features, while some others may have almost no cross-correlations with quite diverse features; and (2) the cross-correlation structure may also change over time due to the system evolution. With this regard, this article presents a dynamic subspace learning method for multivariate functional data modeling. In particular, we consider that different functions come from different subspaces, and only functions of the same subspace have cross-correlations with each other. The subspaces can be automatically formulated and learned by reformatting the problem as a sparse regression. By allowing but regularizing the regression change over time, we can describe the cross-correlation dynamics. The model can be efficiently estimated by the fast iterative shrinkage-thresholding algorithm, and the features of each subspace can be extracted using the smooth multi-channel functional principal component analysis. Some theoretical properties of the model are presented. Numerical studies, together with case studies, demonstrate the efficiency and applicability of the proposed methodology.
Principal Components Analysis in Clinical Studies

Zhongheng Zhang,Adela Castello

DOI: https://doi.org/10.21037/atm.2017.07.12

IF: 3.616

2017-01-01

Annals of Translational Medicine

Abstract:In multivariate analysis, independent variables are usually correlated to each other which can introduce multicollinearity in the regression models. One approach to solve this problem is to apply principal components analysis (PCA) over these variables. This method uses orthogonal transformation to represent sets of potentially correlated variables with principal components (PC) that are linearly uncorrelated. PCs are ordered so that the first PC has the largest possible variance and only some components are selected to represent the correlated variables. As a result, the dimension of the variable space is reduced. This tutorial illustrates how to perform PCA in R environment, the example is a simulated dataset in which two PCs are responsible for the majority of the variance in the data. Furthermore, the visualization of PCA is highlighted.
Multivariate pattern analysis: a method and software to reveal, quantify, and visualize predictive association patterns in multicollinear data

Tim U. H. Baumeister,Eivind Aadland,Roger G. Linington,Olav M. Kvalheim

DOI: https://doi.org/10.1186/s12859-024-05660-6

IF: 3.307

2024-02-02

BMC Bioinformatics

Abstract:Strongly multicollinear covariates, such as those typically represented in metabolomics applications, represent a challenge for multivariate regression analysis. These challenges are commonly circumvented by reducing the number of covariates to a subset of linearly independent variables, but this strategy may lead to loss of resolution and thus produce models with poorer interpretative potential. The aim of this work was to implement and illustrate a method, multivariate pattern analysis (MVPA), which can handle multivariate covariates without compromising resolution or model quality.

biochemical research methods,biotechnology & applied microbiology,mathematical & computational biology
Multivariate Multilinear Regression

Ya Su,Xinbo Gao,Xuelong Li,Dacheng Tao

DOI: https://doi.org/10.1109/tsmcb.2012.2195171

2012-01-01

IEEE Transactions on Systems Man and Cybernetics Part B (Cybernetics)

Abstract:Conventional regression methods, such as multivariate linear regression (MLR) and its extension principal component regression (PCR), deal well with the situations that the data are of the form of low-dimensional vector. When the dimension grows higher, it leads to the under sample problem (USP): the dimensionality of the feature space is much higher than the number of training samples. However, little attention has been paid to such a problem. This paper first adopts an in-depth investigation to the USP in PCR, which answers three questions: 1) Why is USP produced? 2) What is the condition for USP, and 3) How is the influence of USP on regression. With the help of the above analysis, the principal components selection problem of PCR is presented. Subsequently, to address the problem of PCR, a multivariate multilinear regression (MMR) model is proposed which gives a substitutive solution to MLR, under the condition of multilinear objects. The basic idea of MMR is to transfer the multilinear structure of objects into the regression coefficients as a constraint. As a result, the regression problem is reduced to find two low-dimensional coefficients so that the principal components selection problem is avoided. Moreover, the sample size needed for solving MMR is greatly reduced so that USP is alleviated. As there is no closed-form solution for MMR, an alternative projection procedure is designed to obtain the regression matrices. For the sake of completeness, the analysis of computational cost and the proof of convergence are studied subsequently. Furthermore, MMR is applied to model the fitting procedure in the active appearance model (AAM). Experiments are conducted on both the carefully designed synthesizing data set and AAM fitting databases verified the theoretical analysis.
Multivariate Matérn Models -- A Spectral Approach

Drew Yarger,Stilian Stoev,Tailen Hsing

2024-06-02

Abstract:The classical Matérn model has been a staple in spatial statistics. Novel data-rich applications in environmental and physical sciences, however, call for new, flexible vector-valued spatial and space-time models. Therefore, the extension of the classical Matérn model has been a problem of active theoretical and methodological interest. In this paper, we offer a new perspective to extending the Matérn covariance model to the vector-valued setting. We adopt a spectral, stochastic integral approach, which allows us to address challenging issues on the validity of the covariance structure and at the same time to obtain new, flexible, and interpretable models. In particular, our multivariate extensions of the Matérn model allow for asymmetric covariance structures. Moreover, the spectral approach provides an essentially complete flexibility in modeling the local structure of the process. We establish closed-form representations of the cross-covariances when available, compare them with existing models, simulate Gaussian instances of these new processes, and demonstrate estimation of the model's parameters through maximum likelihood. An application of the new class of multivariate Matérn models to environmental data indicate their success in capturing inherent covariance-asymmetry phenomena.

Methodology,Statistics Theory,Other Statistics
Methods and Software for the Multilevel Social Relations Model: A Tutorial

Jeremy Koster,George Leckie,Brandy Aven,Christopher Charlton

2019-07-22

Abstract:This tutorial demonstrates the estimation and interpretation of the Multilevel Social Relations Model for dyadic data. The Social Relations Model is appropriate for data structures in which individuals appear multiple times as both the source and recipient of dyadic outcomes. Estimated using Stat-JR statistical software, the models are fitted to multiple outcome types: continuous, count, and binary outcomes. In addition, models are demonstrated for dyadic data from a single group and from multiple groups. The modeling approaches are illustrated via a series of case studies, and the data and software to replicate these analyses are available as supplemental files.

Methodology
Dependent Matérn Processes for Multivariate Time Series

Alexander Vandenberg-Rodes,Babak Shahbaba

DOI: https://doi.org/10.48550/arXiv.1502.03466

2015-02-12

Abstract:For the challenging task of modeling multivariate time series, we propose a new class of models that use dependent Matérn processes to capture the underlying structure of data, explain their interdependencies, and predict their unknown values. Although similar models have been proposed in the econometric, statistics, and machine learning literature, our approach has several advantages that distinguish it from existing methods: 1) it is flexible to provide high prediction accuracy, yet its complexity is controlled to avoid overfitting; 2) its interpretability separates it from black-box methods; 3) finally, its computational efficiency makes it scalable for high-dimensional time series. In this paper, we use several simulated and real data sets to illustrate these advantages. We will also briefly discuss some extensions of our model.

Machine Learning
Meeting Student Needs for Multivariate Data Analysis: A Case Study in Teaching a Multivariate Data Analysis Course with No Pre-requisites

Amy S. Wagaman

DOI: https://doi.org/10.48550/arXiv.1310.7141

2013-10-26

Other Statistics

Abstract:Modern students encounter big, messy data sets long before setting foot in our classrooms. Many of our students need to develop skills in exploratory data analysis and multivariate analysis techniques for their jobs after college, but these topics are not covered in introductory statistics courses. This case study describes my experience in designing and teaching a course on multivariate data analysis with no pre-requisites, using real data, active learning, and other activities to help students tackle the material.
Fault Detection Techniques Based on Multivariate Statistical Analysis

JI Hongquan,HE Xiao,ZHOU Donghua

DOI: https://doi.org/10.16183/j.cnki.jsjtu.2015.06.018

2015-01-01

Abstract:As an important branch of data-driven fault detection methods,multivariate statistical analysisbased fault detection methods mainly include principal component analysis,partial least squares,independent component analysis and fisher discriminant analysis.In this paper,the data model and fault detection mechanism of each method mentioned above were reviewed.Several properties of these methods were revealed intuitively using simulation results,and their fault detection abilities were illustrated.Finally,several problems related to data-driven fault detection methods were discussed.
Discovering Hidden Knowledge in Data Classification Via Multivariate Analysis

Yisong Chen,Horace H. S. Ip,Sheng Li,Guoping Wang

DOI: https://doi.org/10.1111/j.1468-0394.2009.00508.x

IF: 3.3

2010-01-01

Expert Systems

Abstract:Abstract: A new classification algorithm based on multivariate analysis is proposed to discover and simulate the grading policy on school transcript data sets. The framework comprises three major steps. First, factor analysis is adopted to separate the scores of several different subjects into grading‐related ones and grading‐unrelated ones. Second, multidimensional scaling is employed for dimensionality reduction to facilitate subsequent data visualization and interpretation. Finally, a support vector machine is trained to classify the filtered data into different grades. This work provides an attractive framework for intelligent data analysis and decision making. It also exhibits the advantages of high classification accuracy and supports intuitive data interpretation.
Nonparametric Principal Subspace Regression

Yang Zhou,Mark Koudstaal,Dengdeng Yu,Dehan Kong,Fang Yao

2022-01-01

Journal of Machine Learning Research

Abstract:In scientific applications, multivariate observations often come in tandem with temporal or spa-tial covariates, with which the underlying signals vary smoothly. The standard approaches such as principal component analysis and factor analysis neglect the smoothness of the data, while multivariate linear or nonparametric regression fails to leverage the correlation information among multivariate response variables. We propose a novel approach named nonparametric principal subspace regression to overcome these issues. By decoupling the model discrepancy, a simple two-step estimation procedure is introduced, which takes advantage of the low-rank approximation while keeping smooth dynamics. The theoretical property of the proposed pro-cedure is established under an increasing-dimension framework. We demonstrate the favorable performance of our method in comparison with its counterpart, the conventional nonparametric regression, from both theoretical and numerical perspectives.
High-dimensional Factor Model and Its Applications to Statistical Machine Learning

Zhao Chen,Jianqing Fan,Christina Dan Wang

DOI: https://doi.org/10.1360/ssm-2020-0041

2020-01-01

Scientia Sinica Mathematica

Abstract:This paper reviews the recent developments on factor model and its applications to statistical machine learning. The factor model reduces the dimensionality of variables, and provides a low-rank plus sparse structure for the high-dimensional covariance matrices. Therefore, it attracts much attention in high-dimensional data analysis, and has been widely applied in many fields of sciences, engineering, humanities and social sciences, including economics, finance, genomics, neuroscience, machine learning, and so on. We elaborate how to use principal component analysis method to extract latent factors, estimate their associated factor loadings, idiosyncratic components, and their associated covariance matrices. These methods have been proven to effectively cope with the challenges of big data, such as high dimensionality, strong dependence, heavy-tailed variables, and heterogeneity. In addition, we also focus on the role of the factor model in dealing with high-dimensional statistical learning problems such as covariance matrix estimation, model selection, multiple testing, and prediction. Finally, we illustrate the innate relationships between factor models and modern machine learning problems through several applications, including network analysis, matrix completion, ranking, and mixture models.
Partial Least Squares Methods: Partial Least Squares Correlation and Partial Least Square Regression

Hervé Abdi,Lynne J. Williams

DOI: https://doi.org/10.1007/978-1-62703-059-5_23

2012-08-18

Abstract:Partial least square (pls) methods (also sometimes called projection to latent structures) relate the information present in two data tables that collect measurements on the same set of observations. pls methods proceed by deriving latent variables which are (optimal) linear combinations of the variables of a data table. When the goal is to find the shared information between two tables, the approach is equivalent to a correlation problem and the technique is then called partial least square correlation (PLSC) (also sometimes called pls-SVD). In this case there are two sets of latent variables (one set per table), and these latent variables are required to have maximal covariance. When the goal is to predict one data table the other one, the technique is then called partial least square regression. In this case there is one set of latent variables (derived from the predictor table) and these latent variables are required to give the best possible prediction. In this paper we present and illustrate PLSC and PLSR and show how these descriptive multivariate analysis techniques can be extended to deal with inferential questions by using cross-validation techniques such as the bootstrap and permutation tests.
Applications of Multilevel Bivariate Models in Multiple Correlation.

Zhiwen Li,Gang Liu,Yuchun Tao

DOI: https://doi.org/10.3969/j.issn.1002-3674.2001.01.003

2001-01-01

Abstract:Objective　To analyze the partial correlation problems by multilevel bivariate models.Methods　Multilevel bivariate models.Results　The results of correlation analysis with multilevel bivariate models is same with traditional methods,but the former is more quick and simple.Conclusion　Multilevel bivariate models are effective methods in analyzing the partial correlation problems.
One component partial least squares, high dimensional regression, data splitting, and the multitude of models

David J. Olive,Lingling Zhang

DOI: https://doi.org/10.1080/03610926.2024.2303979

2024-01-24

Abstract:This article gives large sample theory for the one component partial least squares estimator, including some hypothesis tests for high dimensional data, under much weaker conditions than those in the literature. Simple theory is also given for some data splitting estimators and the marginal maximum likelihood estimators. It is shown that lasso, one component partial least squares, and ordinary least squares often estimate different population multiple linear regression models. The article also proves that there are often many valid population models for regression methods such as binary regression.
Multivariate Functional Principal Component Analysis for Data Observed on Different (Dimensional) Domains

C. Happ,S. Greven

DOI: https://doi.org/10.1080/01621459.2016.1273115

2016-12-05

Abstract:Existing approaches for multivariate functional principal component analysis are restricted to data on the same one-dimensional interval. The presented approach focuses on multivariate functional data on different domains that may differ in dimension, e.g. functions and images. The theoretical basis for multivariate functional principal component analysis is given in terms of a Karhunen-Loève Theorem. For the practically relevant case of a finite Karhunen-Loève representation, a relationship between univariate and multivariate functional principal component analysis is established. This offers an estimation strategy to calculate multivariate functional principal components and scores based on their univariate counterparts. For the resulting estimators, asymptotic results are derived. The approach can be extended to finite univariate expansions in general, not necessarily orthonormal bases. It is also applicable for sparse functional data or data with measurement error. A flexible R-implementation is available on CRAN. The new method is shown to be competitive to existing approaches for data observed on a common one-dimensional domain. The motivating application is a neuroimaging study, where the goal is to explore how longitudinal trajectories of a neuropsychological test score covary with FDG-PET brain scans at baseline. Supplementary material, including detailed proofs, additional simulation results and software is available online.

Methodology

Multivariate Data Modeling Methods

Maximum Likelihood Estimators in a Two Step Model for PLS

Applied Multivariate Statistical Analysis

Kernel Multivariate Analysis Framework for Supervised Subspace Learning: A Tutorial on Linear and Kernel Multivariate Methods

A Multivariate Approach to Correlation Analysis Based on Random Matrix Theory

Dynamic Multivariate Functional Data Modeling Via Sparse Subspace Learning

Principal Components Analysis in Clinical Studies

Multivariate pattern analysis: a method and software to reveal, quantify, and visualize predictive association patterns in multicollinear data

Multivariate Multilinear Regression

Multivariate Matérn Models -- A Spectral Approach

Methods and Software for the Multilevel Social Relations Model: A Tutorial

Dependent Matérn Processes for Multivariate Time Series

Meeting Student Needs for Multivariate Data Analysis: A Case Study in Teaching a Multivariate Data Analysis Course with No Pre-requisites

Fault Detection Techniques Based on Multivariate Statistical Analysis

Discovering Hidden Knowledge in Data Classification Via Multivariate Analysis

Nonparametric Principal Subspace Regression

High-dimensional Factor Model and Its Applications to Statistical Machine Learning

Partial Least Squares Methods: Partial Least Squares Correlation and Partial Least Square Regression

Applications of Multilevel Bivariate Models in Multiple Correlation.

One component partial least squares, high dimensional regression, data splitting, and the multitude of models

Multivariate Functional Principal Component Analysis for Data Observed on Different (Dimensional) Domains