Limitations of echocardiographic techniques in evaluation of left atrial masses.

Patricia C. Come,M. Riley,J. Markis,M. Malagold

DOI: https://doi.org/10.1016/0002-9149(81)90363-5

1981-11-01

Abstract:

What problem does this paper attempt to address?

Tests for a Multiple-Sample Problem in High Dimensions

Yanfeng Shen,Zhengyan Lin

DOI: https://doi.org/10.1080/03610926.2012.661505

2013-01-01

Abstract:In this article, we consider the problem of testing the hypothesis on mean vectors in multiple-sample problem when the number of observations is smaller than the number of variables. First we propose an independence rule test (IRT) to deal with high-dimensional effects. The asymptotic distributions of IRT under the null hypothesis as well as under the alternative are established when both the dimension and the sample size go to infinity. Next, using the derived asymptotic power of IRT, we propose an adaptive independence rule test (AIRT) that is particularly designed for testing against sparse alternatives. Our AIRT is novel in that it can effectively pick out a few relevant features and reduce the effect of noise accumulation. Real data analysis and Monte Carlo simulations are used to illustrate our proposed methods.
Testing independence in high dimensions with sums of rank correlations

Dennis Leung,Mathias Drton

DOI: https://doi.org/10.48550/arXiv.1501.01732

2016-12-02

Abstract:We treat the problem of testing independence between m continuous variables when m can be larger than the available sample size n. We consider three types of test statistics that are constructed as sums or sums of squares of pairwise rank correlations. In the asymptotic regime where both m and n tend to infinity, a martingale central limit theorem is applied to show that the null distributions of these statistics converge to Gaussian limits, which are valid with no specific distributional or moment assumptions on the data. Using the framework of U-statistics, our result covers a variety of rank correlations including Kendall's tau and a dominating term of Spearman's rank correlation coefficient (rho), but also degenerate U-statistics such as Hoeffding's $D$, or the $\tau^*$ of Bergsma and Dassios (2014). As in the classical theory for U-statistics, the test statistics need to be scaled differently when the rank correlations used to construct them are degenerate U-statistics. The power of the considered tests is explored in rate-optimality theory under Gaussian equicorrelation alternatives as well as in numerical experiments for specific cases of more general alternatives.

Statistics Theory
The Asymptotic Distribution and Berry–Esseen Bound of a New Test for Independence in High Dimension with an Application to Stochastic Optimization

Wei-Dong Liu,Zhengyan Lin,Qi-Man Shao

DOI: https://doi.org/10.1214/08-aap527

2008-01-01

Abstract:Let X-1 , . . . , X-n be a random sample from a p-dimensional population distribution. Assume that c(1)n(alpha) <= p <= c(2)n(alpha) for some positive constants c(1), c(2) and alpha. In this paper we introduce a new statistic for testing independence of the p-variates of the population and prove that the limiting distribution is the extreme distribution of type I with a rate of convergence O ((log n)(5/2)/root n). This is much faster than O (1/log n), a typical convergence rate for this type of extreme distribution. A simulation study and application to stochastic optimization are discussed.
A hypothesis test for independence of sets of variates in high dimensions

Zhengyan Lin,Yanbiao Xiang

DOI: https://doi.org/10.1016/j.spl.2008.05.003

IF: 0.718

2008-01-01

Statistics & Probability Letters

Abstract:Srivastava [Srivastava, M.S., 2005. Some tests concerning the covariance matrix in high dimensional data. J. Japan Statist. Soc. 35, 251–272] has proposed a test statistic for testing the hypothesis that the covariance matrix of the normal population is a diagonal matrix when the sample size is smaller than the dimensionality of the data. We extend his results to the hypothesis testing problem for independence of sets of variates in high dimensions. A test statistic is proposed and its asymptotic null distribution is also given, as both the sample size and the number of variables go to infinity. Consequently, this test can be used when the number of variables is not small relative to the sample size, in particular, even when the number of variables exceeds the sample size. Simulations are performed to see the accuracy of the asymptotic null distribution.
Distribution-free tests of multivariate independence based on center-outward quadrant, Spearman, Kendall, and van der Waerden statistics

Hongjian Shi,Mathias Drton,Marc Hallin,Fang Han

2024-09-10

Abstract:Due to the lack of a canonical ordering in ${\mathbb R}^d$ for $d>1$, defining multivariate generalizations of the classical univariate ranks has been a long-standing open problem in statistics. Optimal transport has been shown to offer a solution in which multivariate ranks are obtained by transporting data points to a grid that approximates a uniform reference measure (Chernozhukov et al., 2017; Hallin, 2017; Hallin et al., 2021), thereby inducing ranks, signs, and a data-driven ordering of ${\mathbb R}^d$. We take up this new perspective to define and study multivariate analogues of the sign covariance/quadrant statistic, Spearman's rho, Kendall's tau, and van der Waerden covariances. The resulting tests of multivariate independence are fully distribution-free, hence uniformly valid irrespective of the actual (absolutely continuous) distribution of the observations. Our results provide the asymptotic distribution theory for these new test statistics, with asymptotic approximations to critical values to be used for testing independence between random vectors, as well as a power analysis of the resulting tests in an extension of the so-called (bivariate) Konijn model. This power analysis includes a multivariate Chernoff--Savage property guaranteeing that, under elliptical generalized Konijn models, the asymptotic relative efficiency of our van der Waerden tests with respect to Wilks' classical (pseudo-)Gaussian procedure is strictly larger than or equal to one, where equality is achieved under Gaussian distributions only. We similarly provide a lower bound for the asymptotic relative efficiency of our Spearman procedure with respect to Wilks' test, thus extending the classical result by Hodges and Lehmann on the asymptotic relative efficiency, in univariate location models, of Wilcoxon tests with respect to the Student ones.

Statistics Theory
Identity Tests for High Dimensional Data Using Rmt

Cheng Wang,Jing Yang,Baiqi Miao,Longbing Cao

DOI: https://doi.org/10.1016/j.jmva.2013.03.015

2013-01-01

Abstract:In this work, we redefined two important statistics, the CLRT test [Z. Bai, D. Jiang, J. Yao, S. Zheng, Corrections to LRT on large-dimensional covariance matrix by RMT, The Annals of Statistics 37 (6B) (2009) 3822–3840] and the LW test [O. Ledoit, M. Wolf, Some hypothesis tests for the covariance matrix when the dimension is large compared to the sample size, The Annals of Statistics (2002) 1081–1102] on identity tests for high dimensional data using random matrix theories. Compared with existing CLRT and LW tests, the new tests can accommodate data which has unknown means and non-Gaussian distributions. Simulations demonstrate that the new tests have good properties in terms of size and power. What is more, even for Gaussian data, our new tests perform favorably in comparison to existing tests. Finally, we find the CLRT is more sensitive to eigenvalues less than 1 while the LW test has more advantages in relation to detecting eigenvalues larger than 1.
Distance-based and RKHS-based dependence metrics in high dimension

Changbo Zhu,Xianyang Zhang,Shun Yao,Xiaofeng Shao

DOI: https://doi.org/10.1214/19-aos1934

2020-12-01

The Annals of Statistics

Abstract:In this paper, we study distance covariance, Hilbert–Schmidt covariance (aka Hilbert–Schmidt independence criterion [In Advances in Neural Information Processing Systems (2008) 585–592]) and related independence tests under the high dimensional scenario. We show that the sample distance/Hilbert–Schmidt covariance between two random vectors can be approximated by the sum of squared componentwise sample cross-covariances up to an asymptotically constant factor, which indicates that the standard distance/Hilbert–Schmidt covariance based test can only capture linear dependence in high dimension. Under the assumption that the components within each high dimensional vector are weakly dependent, the distance correlation based $t$ test developed by Székely and Rizzo (J. Multivariate Anal. 117 (2013) 193–213) for independence is shown to have trivial limiting power when the two random vectors are nonlinearly dependent but component-wisely uncorrelated. This new and surprising phenomenon, which seems to be discovered and carefully studied for the first time, is further confirmed in our simulation study. As a remedy, we propose tests based on an aggregation of marginal sample distance/Hilbert–Schmidt covariances and show their superior power behavior against their joint counterparts in simulations. We further extend the distance correlation based $t$ test to those based on Hilbert–Schmidt covariance and marginal distance/Hilbert–Schmidt covariance. A novel unified approach is developed to analyze the studentized sample distance/Hilbert–Schmidt covariance as well as the studentized sample marginal distance covariance under both null and alternative hypothesis. Our theoretical and simulation results shed light on the limitation of distance/Hilbert–Schmidt covariance when used jointly in the high dimensional setting and suggest the aggregation of marginal distance/Hilbert–Schmidt covariance as a useful alternative.
On Ranking-based Tests of Independence

Myrto Limnios,Stéphan Clémençon

2024-03-12

Abstract:In this paper we develop a novel nonparametric framework to test the independence of two random variables $\mathbf{X}$ and $\mathbf{Y}$ with unknown respective marginals $H(dx)$ and $G(dy)$ and joint distribution $F(dx dy)$, based on {\it Receiver Operating Characteristic} (ROC) analysis and bipartite ranking. The rationale behind our approach relies on the fact that, the independence hypothesis $\mathcal{H}_0$ is necessarily false as soon as the optimal scoring function related to the pair of distributions $(H\otimes G,\; F)$, obtained from a bipartite ranking algorithm, has a ROC curve that deviates from the main diagonal of the unit square.We consider a wide class of rank statistics encompassing many ways of deviating from the diagonal in the ROC space to build tests of independence. Beyond its great flexibility, this new method has theoretical properties that far surpass those of its competitors. Nonasymptotic bounds for the two types of testing errors are established. From an empirical perspective, the novel procedure we promote in this paper exhibits a remarkable ability to detect small departures, of various types, from the null assumption $\mathcal{H}_0$, even in high dimension, as supported by the numerical experiments presented here.

Statistics Theory,Methodology,Machine Learning
An adaptive test of the independence of high-dimensional data based on Kendall rank correlation coefficient

Lijuan Han,Yun Ma,Junshan Xie

DOI: https://doi.org/10.1080/10485252.2024.2435852

2024-12-06

Journal of Nonparametric Statistics

Abstract:The paper considers the test on the components independence of high-dimensional continuous random vectors based on the Kendall rank correlation coefficient. Under the null hypothesis, it proves that the sum-of-powers type statistics constructed by the Kendall rank correlation coefficient are asymptotically normally distributed and independent when the sum of the orders is odd, and they are also independent with the extreme-value-type test statistics of the Kendall rank correlation coefficient. Based on the asymptotic independence property, it proposes an adaptive test procedure which combines p -values from the sum-of-powers-type test and extreme-value-type test. Some simulations reveal that the proposed test method can significantly enhance the statistical power of the test while keep the type I error well controlled, and it is powerful against a wide range of alternatives (including dense, sparse or moderately sparse signals). To illustrate the use of the proposed adaptive test method, two real data sets are also analysed.

statistics & probability
Testing the independence of two random vectors where only one dimension is large

Weiming Li,Jiaqi Chen,Jianfeng Yao

DOI: https://doi.org/10.1080/02331888.2016.1266988

2015-04-20

Abstract:For testing the independence of two vectors with respective dimensions $p_1$ and $p_2$, the existing literature in high-dimensional statistics all assume that both dimensions $p_1$ and $p_2$ grow to infinity with the sample size. However, as evidenced in the RNA-sequencing data analysis discussed in the paper, it happens frequently that one of the dimension is quite small and the other quite large compared to the sample size. In this paper, we address this new asymptotic framework for the independence test. A new test procedure is introduced and its asymptotic normality is established when the vectors are normal distributed. A Mote-Carlo study demonstrates the consistency of the procedure and exhibits its superiority over some existing high-dimensional procedures. Applied to the RNA-sequencing data mentioned above, we obtain very convincing results on pairwise independence/dependence of gene isoform expressions as attested by prior knowledge established in that field. Lastly, Monte-Carlo experiments show that the procedure is robust against the normality assumption on the population vectors.

Methodology
On high-dimensional tests for mutual independence based on Pearson’s correlation coefficient

Guangyu Mao

DOI: https://doi.org/10.1080/03610926.2019.1593459

2019-03-25

Abstract:This paper investigates a class of statistics based on Pearson’s correlation coefficient for testing the mutual independence of a random vector in high dimensions. Two existing statistics, proposed by Schott (2005) and Mao (2014) respectively, are special cases of the class. A generic testing theory for the class of statistics is developed, which clarifies under what conditions the class of statistics can be employed for the testing purpose. By virtue of the theory, three new tests are introduced, and related statistical properties are discussed. To examine our theoretical findings and check the performance of the new tests, simulation studies are applied. The simulation results justify the theoretical findings and show that the newly introduced tests perform well, as long as both the dimension and the sample size of the data are moderately large.
Distance Correlation Test for High-Dimensional Independence

Weiming Li,Qinwen Wang,Jianfeng Yao

DOI: https://doi.org/10.3150/23-bej1710

IF: 1.5

2024-01-01

Bernoulli

Abstract:In this paper, a new self-normalized and scale invariant statistic Tn, which is based on distance correlations, is developed for testing mutual independence of a high-dimensional random vector. The asymptotic normality of the statistic is established under mild moment conditions by assuming both the dimension p of the vector and the sample size n grow to infinity. In particular, the test procedure has the consistency against sparse alternatives where the dependence can be nonlinear and non-monotonic. Technically, the asymptotic normality of the test statistic is built upon martingale decomposition and novel moment method with appropriate combinatorics.
Penalized Independence Rule for Testing High-Dimensional Hypotheses

Yanfeng Shen,Zhengyan Lin,Jun Zhu

DOI: https://doi.org/10.1080/03610926.2010.484160

2011-01-01

Abstract:We consider a challenging problem of testing any possible association between a response variable and a set of predictors, when the dimensionality of predictors is much greater than the number of observations. In the context of generalized linear models, a new approach is proposed for testing against high-dimensional alternatives. Our method uses soft-thresholding to suppress stochastic noise and applies the independence rule to borrow strength across the predictors. Moreover, the method can provide a ranked predictor list and automatically select "important" features to retain in the test statistic. We compare the performance of this method with some competing approaches via real data and simulation studies, demonstrating that our method maintains relatively higher power against a wide family of alternatives.
Log-Rank-Type Tests for Equality of Distributions in High-Dimensional Spaces.

Linxi Liu,Yang Meng,Xiaoru Wu,Zhiliang Ying,Tian Zheng

DOI: https://doi.org/10.1080/10618600.2022.2051530

2022-01-01

Journal of Computational and Graphical Statistics

Abstract:Motivated by applications in high-dimensional settings, we propose a novel approach for testing the equality of two or more populations by constructing a class of intensity centered score processes. The resulting tests are analogous in spirit to the well-known class of weighted log-rank statistics that are widely used in survival analysis. The test statistics are nonparametric, computationally simple, and applicable to high-dimensional data. We establish the usual large sample properties by showing that the underlying log-rank score process converges weakly to a Gaussian random field with zero mean under the null hypothesis and with a drift under the contiguous alternatives. For the Kolmogorov-Smirnov-type and the Cramervon Mises-type statistics, we also establish the consistency result for any fixed alternative. Cutoff points for the test statistics are obtained by permutations or a simulation-based resampling method. The new approach is applied to a study of brain activation measured by functional magnetic resonance imaging when performing two linguistic tasks and also to a prostate cancer DNA microarray dataset. Supplementary materials for this article are available online.
A Conditional Distribution Function-Based Measure for Independence and K-sample Tests in Multivariate Data

Li Wang,Hongyi Zhou,Weidong Ma,Ying Yang

DOI: https://doi.org/10.1016/j.jmva.2024.105378

IF: 1.387

2025-01-01

Journal of Multivariate Analysis

Abstract:We introduce a new index to measure the degree of dependence and test for independence between two random vectors. The index is obtained by generalizing the Cram & eacute;r-von Mises distances between the conditional and marginal distribution functions via the projection- averaging technique. If one of the random vectors is categorical with K categories, we propose slicing estimators to estimate our index. We conduct an asymptotic analysis for the slicing estimators, considering both situations where K is fixed and where K is allowed to increase with the sample size. When both random vectors are continuous, we introduce a kernel regression estimator for the proposed index, demonstrating that its asymptotic null distribution follows a normal distribution and conducting a local power analysis for the kernel estimator-based independence test. The proposed tests are studied via simulations, with a real data application presented to illustrate our methods.
Reducing multivariate independence testing to two bivariate means comparisons

Kai Xu,Yeqing Zhou,Liping Zhu,Runze Li

2024-02-25

Abstract:Testing for independence between two random vectors is a fundamental problem in statistics. It is observed from empirical studies that many existing omnibus consistent tests may not work well for some strongly nonmonotonic and nonlinear relationships. To explore the reasons behind this issue, we novelly transform the multivariate independence testing problem equivalently into checking the equality of two bivariate means. An important observation we made is that the power loss is mainly due to cancellation of positive and negative terms in dependence metrics, making them very close to zero. Motivated by this observation, we propose a class of consistent metrics with a positive integer $\gamma$ that exactly characterize independence. Theoretically, we show that the metrics with even and infinity $\gamma$ can effectively avoid the cancellation, and have high powers under the alternatives that two mean differences offset each other. Since we target at a wide range of dependence scenarios in practice, we further suggest to combine the p-values of test statistics with different $\gamma$'s through the Fisher's method. We illustrate the advantages of our proposed tests through extensive numerical studies.

Methodology
Asymptotic Distribution-Free Independence Test for High Dimension Data

Zhanrui Cai,Jing Lei,Kathryn Roeder

DOI: https://doi.org/10.48550/arXiv.2110.07652

2023-06-11

Abstract:Test of independence is of fundamental importance in modern data analysis, with broad applications in variable selection, graphical models, and causal inference. When the data is high dimensional and the potential dependence signal is sparse, independence testing becomes very challenging without distributional or structural assumptions. In this paper, we propose a general framework for independence testing by first fitting a classifier that distinguishes the joint and product distributions, and then testing the significance of the fitted classifier. This framework allows us to borrow the strength of the most advanced classification algorithms developed from the modern machine learning community, making it applicable to high dimensional, complex data. By combining a sample split and a fixed permutation, our test statistic has a universal, fixed Gaussian null distribution that is independent of the underlying data distribution. Extensive simulations demonstrate the advantages of the newly proposed test compared with existing methods. We further apply the new test to a single-cell data set to test the independence between two types of single-cell sequencing measurements, whose high dimensionality and sparsity make existing methods hard to apply.

Methodology
Rank Based Tests for High Dimensional White Noise

Dachuan Chen,Fengyi Song,Long Feng

2023-07-19

Abstract:The development of high-dimensional white noise test is important in both statistical theories and applications, where the dimension of the time series can be comparable to or exceed the length of the time series. This paper proposes several distribution-free tests using the rank based statistics for testing the high-dimensional white noise, which are robust to the heavy tails and do not quire the finite-order moment assumptions for the sample distributions. Three families of rank based tests are analyzed in this paper, including the simple linear rank statistics, non-degenerate U-statistics and degenerate U-statistics. The asymptotic null distributions and rate optimality are established for each family of these tests. Among these tests, the test based on degenerate U-statistics can also detect the non-linear and non-monotone relationships in the autocorrelations. Moreover, this is the first result on the asymptotic distributions of rank correlation statistics which allowing for the cross-sectional dependence in high dimensional data.

Statistics Theory,Methodology
A distribution-free test of independence based on a modified mean variance index

Weidong Ma,Fei Ye,Jingsong Xiao,Ying Yang

DOI: https://doi.org/10.1080/24754269.2023.2201101

2023-04-29

Statistical Theory and Related Fields

Abstract:Cui and Zhong (2019), ( Computational Statistics & Data Analysis , 139, 117–133) proposed a test based on the mean variance (MV) index to test independence between a categorical random variable Y with R categories and a continuous random variable X . They ingeniously proved the asymptotic normality of the MV test statistic when R diverges to infinity, which brings many merits to the MV test, including making it more convenient for independence testing when R is large. This paper considers a new test called the integral Pearson chi-square (IPC) test, whose test statistic can be viewed as a modified MV test statistic. A central limit theorem of the martingale difference is used to show that the asymptotic null distribution of the standardized IPC test statistic when R is diverging is also a normal distribution, rendering the IPC test sharing many merits with the MV test. As an application of such a theoretical finding, the IPC test is extended to test independence between continuous random variables. The finite sample performance of the proposed test is assessed by Monte Carlo simulations, and a real data example is presented for illustration.
Use of Random Integration to Test Equality of High Dimensional Covariance Matrices

Yunlu Jiang,Canhong Wen,Yukang Jiang,Xueqin Wang,Heping Zhang

DOI: https://doi.org/10.5705/ss.202020.0486

IF: 1.4

2024-01-01

Statistica Sinica

Abstract:Testing the equality of two covariance matrices is a fundamental problem in statistics, and especially challenging when the data are high-dimensional. Through a novel use of random integration, we can test the equality of high-dimensional covariance matrices without assuming parametric distributions for the two underlying populations, even if the dimension is much larger than the sample size. The asymptotic properties of our test for arbitrary number of covariates and sample size are studied in depth under a general multivariate model. The finite-sample performance of our test is evaluated through numerical studies. The empirical results demonstrate that our test is highly competitive with existing tests in a wide range of settings. In particular, our proposed test is distinctly powerful under different settings when there exist a few large or many small diagonal disturbances between the two covariance matrices.

statistics & probability

Limitations of echocardiographic techniques in evaluation of left atrial masses.

Tests for a Multiple-Sample Problem in High Dimensions

Testing independence in high dimensions with sums of rank correlations

The Asymptotic Distribution and Berry–Esseen Bound of a New Test for Independence in High Dimension with an Application to Stochastic Optimization

A hypothesis test for independence of sets of variates in high dimensions

Distribution-free tests of multivariate independence based on center-outward quadrant, Spearman, Kendall, and van der Waerden statistics

Identity Tests for High Dimensional Data Using Rmt

Distance-based and RKHS-based dependence metrics in high dimension

On Ranking-based Tests of Independence

An adaptive test of the independence of high-dimensional data based on Kendall rank correlation coefficient

Testing the independence of two random vectors where only one dimension is large

On high-dimensional tests for mutual independence based on Pearson’s correlation coefficient

Distance Correlation Test for High-Dimensional Independence

Penalized Independence Rule for Testing High-Dimensional Hypotheses

Log-Rank-Type Tests for Equality of Distributions in High-Dimensional Spaces.

A Conditional Distribution Function-Based Measure for Independence and K-sample Tests in Multivariate Data

Reducing multivariate independence testing to two bivariate means comparisons

Asymptotic Distribution-Free Independence Test for High Dimension Data

Rank Based Tests for High Dimensional White Noise

A distribution-free test of independence based on a modified mean variance index

Use of Random Integration to Test Equality of High Dimensional Covariance Matrices