De novo and comparative transcriptome analysis of cultivated and wild spinach

Chenxi Xu,C. Jiao,Yi Zheng,Honghe Sun,Wenli Liu,Xiaofeng Cai,Xiaoli Wang,Shuang Liu,Yimin Xu,B. Mou,Shaojun Dai,Z. Fei,Quanhua Wang

DOI: https://doi.org/10.1038/srep17706

IF: 4.6

2015-12-04

Scientific Reports

Abstract:

What problem does this paper attempt to address?

A hypothesis test for independence of sets of variates in high dimensions

Zhengyan Lin,Yanbiao Xiang

DOI: https://doi.org/10.1016/j.spl.2008.05.003

IF: 0.718

2008-01-01

Statistics & Probability Letters

Abstract:Srivastava [Srivastava, M.S., 2005. Some tests concerning the covariance matrix in high dimensional data. J. Japan Statist. Soc. 35, 251–272] has proposed a test statistic for testing the hypothesis that the covariance matrix of the normal population is a diagonal matrix when the sample size is smaller than the dimensionality of the data. We extend his results to the hypothesis testing problem for independence of sets of variates in high dimensions. A test statistic is proposed and its asymptotic null distribution is also given, as both the sample size and the number of variables go to infinity. Consequently, this test can be used when the number of variables is not small relative to the sample size, in particular, even when the number of variables exceeds the sample size. Simulations are performed to see the accuracy of the asymptotic null distribution.
High dimensional correlation matrices: the central limit theorem and its applications

Jiti Gao,Xiao Han,Guangming Pan,Yanrong Yang

DOI: https://doi.org/10.1111/rssb.12189

2016-06-24

Abstract:Summary Statistical inferences for sample correlation matrices are important in high dimensional data analysis. Motivated by this, the paper establishes a new central limit theorem for a linear spectral statistic of high dimensional sample correlation matrices for the case where the dimension p and the sample size n are comparable. This result is of independent interest in large dimensional random-matrix theory. We also further investigate the sample correlation matrices of a high dimensional vector whose elements have a special correlated structure and the corresponding central limit theorem is developed. Meanwhile, we apply the linear spectral statistic to an independence test for p random variables, and then an equivalence test for p factor loadings and n factors in a factor model. The finite sample performance of the test proposed shows its applicability and effectiveness in practice. An empirical application to test the independence of household incomes from various cities in China is also conducted.

statistics & probability
Generalized kernel distance covariance in high dimensions: non-null CLTs and power universality

Qiyang Han,Yandi Shen

2024-08-02

Abstract:Distance covariance is a popular dependence measure for two random vectors $X$ and $Y$ of possibly different dimensions and types. Recent years have witnessed concentrated efforts in the literature to understand the distributional properties of the sample distance covariance in a high-dimensional setting, with an exclusive emphasis on the null case that $X$ and $Y$ are independent. This paper derives the first non-null central limit theorem for the sample distance covariance, and the more general sample (Hilbert-Schmidt) kernel distance covariance in high dimensions, primarily in the Gaussian case. The new non-null central limit theorem yields an asymptotically exact first-order power formula for the widely used generalized kernel distance correlation test of independence between $X$ and $Y$. The power formula in particular unveils an interesting universality phenomenon: the power of the generalized kernel distance correlation test is completely determined by $n\cdot \text{dcor}^2(X,Y)/\sqrt{2}$ in the high dimensional limit, regardless of a wide range of choices of the kernels and bandwidth parameters. Furthermore, this separation rate is also shown to be optimal in a minimax sense. The key step in the proof of the non-null central limit theorem is a precise expansion of the mean and variance of the sample distance covariance in high dimensions, which shows, among other things, that the non-null Gaussian approximation of the sample distance covariance involves a rather subtle interplay between the dimension-to-sample ratio and the dependence between $X$ and $Y$.

Statistics Theory
The Asymptotic Distribution and Berry–Esseen Bound of a New Test for Independence in High Dimension with an Application to Stochastic Optimization

Wei-Dong Liu,Zhengyan Lin,Qi-Man Shao

DOI: https://doi.org/10.1214/08-aap527

2008-01-01

Abstract:Let X-1 , . . . , X-n be a random sample from a p-dimensional population distribution. Assume that c(1)n(alpha) <= p <= c(2)n(alpha) for some positive constants c(1), c(2) and alpha. In this paper we introduce a new statistic for testing independence of the p-variates of the population and prove that the limiting distribution is the extreme distribution of type I with a rate of convergence O ((log n)(5/2)/root n). This is much faster than O (1/log n), a typical convergence rate for this type of extreme distribution. A simulation study and application to stochastic optimization are discussed.
Detecting practically significant dependencies in infinite dimensional data via distance correlations

Holger Dette,Marius Kroll

2024-11-25

Abstract:In this paper we take a different look on the problem of testing the independence of two infinite dimensional random variables using the distance correlation. Instead of testing if the distance correlation vanishes exactly, we are interested in the null hypothesis that it does not exceed a certain threshold. Our formulation of the testing problem is motivated by the observation that in many cases it is more reasonable to test for a practically significant dependency since it is rare that a null hypothesis of perfect independence is exactly satisfied. This point of view also reflects statistical practice, where one often classifies the strength of the association in categories such as 'small', 'medium' and 'large' and the precise definitions depend on the specific application. To address these problems we develop a pivotal test for the hypothesis that the distance correlation $\mathrm{dcor}(X,Y)$ between two random variables $X$ and $Y$ does not exceed a pre-specified threshold $\Delta$, that is $H_0 : \mathrm{dcor}(X,Y) \leq \Delta$ versus $H_1 : \mathrm{dcor}(X,Y) > \Delta$. We also determine a minimum value $\hat \Delta_\alpha$ from the data such that $H_0$ is rejected for all $\Delta \leq \hat \Delta_\alpha$ at controlled type I error $\alpha$. This quantity can be interpreted as a measure of evidence against the hypothesis of independence. The new test is applicable to data modeled by a strictly stationary and absolutely regular process with components taking values in separable metric spaces of negative type, which includes Euclidean as well as functional data. Our approach is based on a new functional limit theorem for the sequential distance correlation process.

Statistics Theory,Probability
Distance Correlation Test for High-Dimensional Independence

Weiming Li,Qinwen Wang,Jianfeng Yao

DOI: https://doi.org/10.3150/23-bej1710

IF: 1.5

2024-01-01

Bernoulli

Abstract:In this paper, a new self-normalized and scale invariant statistic Tn, which is based on distance correlations, is developed for testing mutual independence of a high-dimensional random vector. The asymptotic normality of the statistic is established under mild moment conditions by assuming both the dimension p of the vector and the sample size n grow to infinity. In particular, the test procedure has the consistency against sparse alternatives where the dependence can be nonlinear and non-monotonic. Technically, the asymptotic normality of the test statistic is built upon martingale decomposition and novel moment method with appropriate combinatorics.
Distance-based and RKHS-based dependence metrics in high dimension

Changbo Zhu,Xianyang Zhang,Shun Yao,Xiaofeng Shao

DOI: https://doi.org/10.1214/19-aos1934

2020-12-01

The Annals of Statistics

Abstract:In this paper, we study distance covariance, Hilbert–Schmidt covariance (aka Hilbert–Schmidt independence criterion [In Advances in Neural Information Processing Systems (2008) 585–592]) and related independence tests under the high dimensional scenario. We show that the sample distance/Hilbert–Schmidt covariance between two random vectors can be approximated by the sum of squared componentwise sample cross-covariances up to an asymptotically constant factor, which indicates that the standard distance/Hilbert–Schmidt covariance based test can only capture linear dependence in high dimension. Under the assumption that the components within each high dimensional vector are weakly dependent, the distance correlation based $t$ test developed by Székely and Rizzo (J. Multivariate Anal. 117 (2013) 193–213) for independence is shown to have trivial limiting power when the two random vectors are nonlinearly dependent but component-wisely uncorrelated. This new and surprising phenomenon, which seems to be discovered and carefully studied for the first time, is further confirmed in our simulation study. As a remedy, we propose tests based on an aggregation of marginal sample distance/Hilbert–Schmidt covariances and show their superior power behavior against their joint counterparts in simulations. We further extend the distance correlation based $t$ test to those based on Hilbert–Schmidt covariance and marginal distance/Hilbert–Schmidt covariance. A novel unified approach is developed to analyze the studentized sample distance/Hilbert–Schmidt covariance as well as the studentized sample marginal distance covariance under both null and alternative hypothesis. Our theoretical and simulation results shed light on the limitation of distance/Hilbert–Schmidt covariance when used jointly in the high dimensional setting and suggest the aggregation of marginal distance/Hilbert–Schmidt covariance as a useful alternative.
Eigenvalue Distribution of a High-Dimensional Distance Covariance Matrix with Application

Weiming Li,Qinwen Wang,Jianfeng Yao

DOI: https://doi.org/10.5705/ss.202020.0327

IF: 1.4

2021-01-01

Statistica Sinica

Abstract:We introduce a new random matrix model called distance covariance matrix in this paper, whose normalized trace is equivalent to the distance covariance. We first derive a deterministic limit for the eigenvalue distribution of the distance covariance matrix when the dimensions of the vectors and the sample size tend to infinity simultaneously. This limit is valid when the vectors are independent or weakly dependent through a finite-rank perturbation. It is also universal and independent of the details of the distributions of the vectors. Furthermore, the top eigenvalues of this distance covariance matrix are shown to obey an exact phase transition when the dependence of the vectors is of finite rank. This finding enables the construction of a new detector for such weak dependence where classical methods based on large sample covariance matrices or sample canonical correlations may fail in the considered high-dimensional framework.
The Chi-Square Test of Distance Correlation

Cencheng Shen,Sambit Panda,Joshua T. Vogelstein

DOI: https://doi.org/10.1080/10618600.2021.1938585

2021-05-15

Abstract:Distance correlation has gained much recent attention in the data science community: the sample statistic is straightforward to compute and asymptotically equals zero if and only if independence, making it an ideal choice to discover any type of dependency structure given sufficient sample size. One major bottleneck is the testing process: because the null distribution of distance correlation depends on the underlying random variables and metric choice, it typically requires a permutation test to estimate the null and compute the p-value, which is very costly for large amount of data. To overcome the difficulty, in this paper we propose a chi-square test for distance correlation. Method-wise, the chi-square test is non-parametric, extremely fast, and applicable to bias-corrected distance correlation using any strong negative type metric or characteristic kernel. The test exhibits a similar testing power as the standard permutation test, and can be utilized for K-sample and partial testing. Theory-wise, we show that the underlying chi-square distribution well approximates and dominates the limiting null distribution in upper tail, prove the chi-square test can be valid and universally consistent for testing independence, and establish a testing power inequality with respect to the permutation test.

Machine Learning,Statistics Theory,Methodology
Distance Correlation in Multiple Biased Sampling Models

Yuwei Ke,Hok Kan Ling,Yanglei Song

2024-08-22

Abstract:Testing the independence between random vectors is a fundamental problem in statistics. Distance correlation, a recently popular dependence measure, is universally consistent for testing independence against all distributions with finite moments. However, when data are subject to selection bias or collected from multiple sources or schemes, spurious dependence may arise. This creates a need for methods that can effectively utilize data from different sources and correct these biases. In this paper, we study the estimation of distance covariance and distance correlation under multiple biased sampling models, which provide a natural framework for addressing these issues. Theoretical properties, including the strong consistency and asymptotic null distributions of the distance covariance and correlation estimators, and the rate at which the test statistic diverges under sequences of alternatives approaching the null, are established. A weighted permutation procedure is proposed to determine the critical value of the independence test. Simulation studies demonstrate that our approach improves both the estimation of distance correlation and the power of the test.

Methodology,Statistics Theory
The affinely invariant distance correlation

Johannes Dueck,Dominic Edelmann,Tilmann Gneiting,Donald Richards

DOI: https://doi.org/10.3150/13-bej558

IF: 1.5

2014-11-01

Bernoulli

Abstract:Székely, Rizzo and Bakirov (Ann. Statist. 35 (2007) 2769–2794) and Székely and Rizzo (Ann. Appl. Statist. 3 (2009) 1236–1265), in two seminal papers, introduced the powerful concept of distance correlation as a measure of dependence between sets of random variables. We study in this paper an affinely invariant version of the distance correlation and an empirical version of that distance correlation, and we establish the consistency of the empirical quantity. In the case of subvectors of a multivariate normally distributed random vector, we provide exact expressions for the affinely invariant distance correlation in both finite-dimensional and asymptotic settings, and in the finite-dimensional case we find that the affinely invariant distance correlation is a function of the canonical correlation coefficients. To illustrate our results, we consider time series of wind vectors at the Stateline wind energy center in Oregon and Washington, and we derive the empirical auto and cross distance correlation functions between wind vectors at distinct meteorological stations.

statistics & probability
Large dimensional Spearman's rank correlation matrices: The central limit theorem and its applications

Hantao Chen,Cheng Wang

2024-11-24

Abstract:This paper is concerned with Spearman's correlation matrices under large dimensional regime, in which the data dimension diverges to infinity proportionally with the sample size. We establish the central limit theorem for the linear spectral statistics of Spearman's correlation matrices, which extends the results of [\emph{Ann. Statist.} 43(2015) 2588--2623]. We also study the improved Spearman's correlation matrices [\emph{Ann. Math. Statist} 19(1948) 293--325] which is a standard U-statistic of order 3. As applications, we propose three new test statistics for large dimensional independent test and numerical studies demonstrate the applicability of our proposed methods.

Statistics Theory
Central limit theorems for high dimensional dependent data

Jinyuan Chang,Xiaohui Chen,Mingcong Wu

DOI: https://doi.org/10.3150/23-BEJ1614

2023-03-24

Abstract:Motivated by statistical inference problems in high-dimensional time series data analysis, we first derive non-asymptotic error bounds for Gaussian approximations of sums of high-dimensional dependent random vectors on hyper-rectangles, simple convex sets and sparsely convex sets. We investigate the quantitative effect of temporal dependence on the rates of convergence to a Gaussian random vector over three different dependency frameworks ($\alpha$-mixing, $m$-dependent, and physical dependence measure). In particular, we establish new error bounds under the $\alpha$-mixing framework and derive faster rate over existing results under the physical dependence measure. To implement the proposed results in practical statistical inference problems, we also derive a data-driven parametric bootstrap procedure based on a kernel estimator for the long-run covariance matrices. We apply the unified Gaussian and bootstrap approximation results to test mean vectors with combined $\ell^2$ and $\ell^\infty$ type statistics, change point detection, and construction of confidence regions for covariance and precision matrices, all for time series data.

Statistics Theory,Probability
Distribution and Correlation Free Two-Sample Test of High-Dimensional Means

Kaijie Xue,Fang Yao

DOI: https://doi.org/10.1214/19-aos1848

2020-01-01

The Annals of Statistics

Abstract:We propose a two-sample test for high-dimensional means that requires neither distributional nor correlational assumptions, besides some weak conditions on the moments and tail properties of the elements in the random vectors. This two-sample test based on a nontrivial extension of the one-sample central limit theorem (Chernozhukov et al., 2017) provides a practically useful procedure with rigorous theoretical guarantees on its size and power assessment. In particular, the proposed test is easy to compute and does not require the independently and identically distributed assumption, which is allowed to have different distributions and arbitrary correlation structures. Further desired features include weaker moments and tail conditions than existing methods, allowance for highly unequal sample sizes, consistent power behavior under fairly general alternative, data dimension allowed to be exponentially high under the umbrella of such general conditions. Simulated and real data examples are used to demonstrate the favorable numerical performance over existing methods.
Uniform-over-dimension convergence with application to location tests for high-dimensional data

Joydeep Chowdhury,Subhajit Dutta,Marc G. Genton

2024-03-25

Abstract:Asymptotic methods for hypothesis testing in high-dimensional data usually require the dimension of the observations to increase to infinity, often with an additional condition on its rate of increase compared to the sample size. On the other hand, multivariate asymptotic methods are valid for fixed dimension only, and their practical implementations in hypothesis testing methodology typically require the sample size to be large compared to the dimension for yielding desirable results. However, in practical scenarios, it is usually not possible to determine whether the dimension of the data at hand conform to the conditions required for the validity of the high-dimensional asymptotic methods, or whether the sample size is large enough compared to the dimension of the data. In this work, a theory of asymptotic convergence is proposed, which holds uniformly over the dimension of the random vectors. This theory attempts to unify the asymptotic results for fixed-dimensional multivariate data and high-dimensional data, and accounts for the effect of the dimension of the data on the performance of the hypothesis testing procedures. The methodology developed based on this asymptotic theory can be applied to data of any dimension. An application of this theory is demonstrated in the two-sample test for the equality of locations. The test statistic proposed is unscaled by the sample covariance, similar to usual tests for high-dimensional data. Using simulated examples, it is demonstrated that the proposed test exhibits better performance compared to several popular tests in the literature for high-dimensional data. Further, it is demonstrated in simulated models that the proposed unscaled test performs better than the usual scaled two-sample tests for multivariate data, including the Hotelling's $T^2$ test for multivariate Gaussian data.

Statistics Theory
On the Decreasing Power of Kernel and Distance Based Nonparametric Hypothesis Tests in High Dimensions

Aaditya Ramdas,Sashank Jakkam Reddi,Barnabas Poczos,Aarti Singh,Larry Wasserman

DOI: https://doi.org/10.1609/aaai.v29i1.9692

2015-03-04

Proceedings of the AAAI Conference on Artificial Intelligence

Abstract:This paper is about two related decision theoretic problems, nonparametric two-sample testing and independence testing. There is a belief that two recently proposed solutions, based on kernels and distances between pairs of points, behave well in high-dimensional settings. We identify different sources of misconception that give rise to the above belief. Specifically, we differentiate the hardness of estimation of test statistics from the hardness of testing whether these statistics are zero or not, and explicitly discuss a notion of "fair" alternative hypotheses for these problems as dimension increases. We then demonstrate that the power of these tests actually drops polynomially with increasing dimension against fair alternatives. We end with some theoretical insights and shed light on the median heuristic for kernel bandwidth selection. Our work advances the current understanding of the power of modern nonparametric hypothesis tests in high dimensions.

English Else
Central Limit Theorem for Linear Spectral Statistics of Large Dimensional Kendall's Rank Correlation Matrices and its Applications

Zeng Li,Qinwen Wang,Runze Li

DOI: https://doi.org/10.48550/arXiv.1912.06357

2019-12-13

Abstract:This paper is concerned with the limiting spectral behaviors of large dimensional Kendall's rank correlation matrices generated by samples with independent and continuous components. We do not require the components to be identically distributed, and do not need any moment conditions, which is very different from the assumptions imposed in the literature of random matrix theory. The statistical setting in this paper covers a wide range of highly skewed and heavy-tailed distributions. We establish the central limit theorem (CLT) for the linear spectral statistics of the Kendall's rank correlation matrices under the Marchenko-Pastur asymptotic regime, in which the dimension diverges to infinity proportionally with the sample size. We further propose three nonparametric procedures for high dimensional independent test and their limiting null distributions are derived by implementing this CLT. Our numerical comparisons demonstrate the robustness and superiority of our proposed test statistics under various mixed and heavy-tailed cases.

Statistics Theory,Methodology
Asymptotic Distribution-Free Independence Test for High Dimension Data

Zhanrui Cai,Jing Lei,Kathryn Roeder

DOI: https://doi.org/10.48550/arXiv.2110.07652

2023-06-11

Abstract:Test of independence is of fundamental importance in modern data analysis, with broad applications in variable selection, graphical models, and causal inference. When the data is high dimensional and the potential dependence signal is sparse, independence testing becomes very challenging without distributional or structural assumptions. In this paper, we propose a general framework for independence testing by first fitting a classifier that distinguishes the joint and product distributions, and then testing the significance of the fitted classifier. This framework allows us to borrow the strength of the most advanced classification algorithms developed from the modern machine learning community, making it applicable to high dimensional, complex data. By combining a sample split and a fixed permutation, our test statistic has a universal, fixed Gaussian null distribution that is independent of the underlying data distribution. Extensive simulations demonstrate the advantages of the newly proposed test compared with existing methods. We further apply the new test to a single-cell data set to test the independence between two types of single-cell sequencing measurements, whose high dimensionality and sparsity make existing methods hard to apply.

Methodology
Spectral properties of high dimensional rescaled sample correlation matrices

Weijiang Chen,Shurong Zheng,Tingting Zou

2024-08-29

Abstract:High-dimensional sample correlation matrices are a crucial class of random matrices in multivariate statistical analysis. The central limit theorem (CLT) provides a theoretical foundation for statistical inference. In this paper, assuming that the data dimension increases proportionally with the sample size, we derive the limiting spectral distribution of the matrix $\widehat{\mathbf{R}}_n\mathbf{M}$ and establish the CLTs for the linear spectral statistics (LSS) of $\widehat{\mathbf{R}}_n\mathbf{M}$ in two structures: linear independent component structure and elliptical structure. In contrast to existing literature, our proposed spectral properties do not require $\mathbf{M}$ to be an identity matrix. Moreover, we also derive the joint limiting distribution of LSSs of $\widehat{\mathbf{R}}_n \mathbf{M}_1,\ldots,\widehat{\mathbf{R}}_n \mathbf{M}_K$. As an illustration, an application is given for the CLT.

Statistics Theory
A unified framework for correlation mining in ultra-high dimension

Yun Wei,Bala Rajaratnam,Alfred O. Hero

DOI: https://doi.org/10.48550/arXiv.2101.04715

2022-08-24

Abstract:Many applications benefit from theory relevant to the identification of variables having large correlations or partial correlations in high dimension. Recently there has been progress in the ultra-high dimensional setting when the sample size $n$ is fixed and the dimension $p$ tends to infinity. Despite these advances, the correlation screening framework suffers from practical, methodological and theoretical deficiencies. For instance, previous correlation screening theory requires that the population covariance matrix be sparse and block diagonal. This block sparsity assumption is however restrictive in practical applications. As a second example, correlation and partial correlation screening requires the estimation of dependence measures, which can be computationally prohibitive. In this paper, we propose a unifying approach to correlation and partial correlation mining that is not restricted to block diagonal correlation structure, thus yielding a methodology that is suitable for modern applications. By making connections to random geometric graphs, the number of highly correlated or partial correlated variables are shown to have compound Poisson finite-sample characterizations, which hold for both the finite $p$ case and when $p$ tends to infinity. The unifying framework also demonstrates a duality between correlation and partial correlation screening with theoretical and practical consequences.

Statistics Theory

De novo and comparative transcriptome analysis of cultivated and wild spinach

A hypothesis test for independence of sets of variates in high dimensions

High dimensional correlation matrices: the central limit theorem and its applications

Generalized kernel distance covariance in high dimensions: non-null CLTs and power universality

The Asymptotic Distribution and Berry–Esseen Bound of a New Test for Independence in High Dimension with an Application to Stochastic Optimization

Detecting practically significant dependencies in infinite dimensional data via distance correlations

Distance Correlation Test for High-Dimensional Independence

Distance-based and RKHS-based dependence metrics in high dimension

Eigenvalue Distribution of a High-Dimensional Distance Covariance Matrix with Application

The Chi-Square Test of Distance Correlation

Distance Correlation in Multiple Biased Sampling Models

The affinely invariant distance correlation

Large dimensional Spearman's rank correlation matrices: The central limit theorem and its applications

Central limit theorems for high dimensional dependent data

Distribution and Correlation Free Two-Sample Test of High-Dimensional Means

Uniform-over-dimension convergence with application to location tests for high-dimensional data

On the Decreasing Power of Kernel and Distance Based Nonparametric Hypothesis Tests in High Dimensions

Central Limit Theorem for Linear Spectral Statistics of Large Dimensional Kendall's Rank Correlation Matrices and its Applications

Asymptotic Distribution-Free Independence Test for High Dimension Data

Spectral properties of high dimensional rescaled sample correlation matrices

A unified framework for correlation mining in ultra-high dimension