Abstract:BACKGROUND:With the increasing amount of data generated in molecular genetics laboratories, it is often difficult to make sense of results because of the vast number of different outcomes or variables studied. Examples include expression levels for large numbers of genes and haplotypes at large numbers of loci. It is then natural to group observations into smaller numbers of classes that allow for an easier overview and interpretation of the data. This grouping is often carried out in multiple steps with the aid of hierarchical cluster analysis, each step leading to a smaller number of classes by combining similar observations or classes. At each step, either implicitly or explicitly, researchers tend to interpret results and eventually focus on that set of classes providing the "best" (most significant) result. While this approach makes sense, the overall statistical significance of the experiment must include the clustering process, which modifies the grouping structure of the data and often removes variation.RESULTS:For hierarchically clustered data, we propose considering the strongest result or, equivalently, the smallest p-value as the experiment-wise statistic of interest and evaluating its significance level for a global assessment of statistical significance. We apply our approach to datasets from haplotype association and microarray expression studies where hierarchical clustering has been used.CONCLUSION:In all of the cases we examine, we find that relying on one set of classes in the course of clustering leads to significance levels that are too small when compared with the significance level associated with an overall statistic that incorporates the process of clustering. In other words, relying on one step of clustering may furnish a formally significant result while the overall experiment is not significant.

Testing for a difference in means of a single feature after clustering

Selective inference for k-means clustering

Selective Inference for Hierarchical Clustering

Post-clustering difference testing: Valid inference and practical considerations with applications to ecological and biological data

Testing for Unobserved Heterogeneity via k-means Clustering

Valid Post-clustering Differential Analysis for Single-Cell RNA-Seq

Simulation-Based Hypothesis Testing of High Dimensional Means Under Covariance Heterogeneity

Testing for a Change in Mean After Changepoint Detection

False Discovery Rate Control via Data Splitting for Testing-after-Clustering

Inference on differences between classes using cluster-specific contrasts of mixed effects

Mistaken identities lead to missed opportunities: Testing for mean differences in partially matched data

Statistical Significance for Hierarchical Clustering in Genetic Association and Microarray Expression Studies.

Significance analysis for clustering with single-cell RNA-sequencing data

Power-Enhanced Two-Sample Mean Tests for High-Dimensional Compositional Data with Application to Microbiome Data Analysis

Testing the significance of patterns in data with cluster structure

Comparison of tests for categorical data from a stratified cluster randomized trial

Feature Screening for Clustering Analysis

Extension of the Dip-test Repertoire -- Efficient and Differentiable p-value Calculation for Clustering

Towards a theory for testing statistical hypothesis: Multivariate mean with nuisance covariance matrix

On testing the equality of high dimensional mean vectors with unequal covariance matrices

On Testing the Mean Equivalence of Treatments from Correlated Normal Populations