Abstract:Clustering methods have led to a number of important discoveries in bioinformatics and beyond. A major challenge in their use is determining which clusters represent important underlying structure, as opposed to spurious sampling artifacts. This challenge is especially serious, and very few methods are available, when the data are very high in dimension. Statistical Significance of Clustering (SigClust) is a recently developed cluster evaluation tool for high dimensional low sample size data. An important component of the SigClust approach is the very definition of a single cluster as a subset of data sampled from a multivariate Gaussian distribution. The implementation of SigClust requires the estimation of the eigenvalues of the covariance matrix for the null multivariate Gaussian distribution. We show that the original eigenvalue estimation can lead to a test that suffers from severe inflation of type-I error, in the important case where there are a few very large eigenvalues. This paper addresses this critical challenge using a novel likelihood based soft thresholding approach to estimate these eigenvalues, which leads to a much improved SigClust. Major improvements in SigClust performance are shown by both mathematical analysis, based on the new notion of Theoretical Cluster Index, and extensive simulation studies. Applications to some cancer genomic data further demonstrate the usefulness of these improvements.

Cluster validation by prediction strength

Powerful Significance Testing for Unbalanced Clusters

A Clustering Approach in Developing Prognostic Systems of Cancer Patients

Developing Prognostic Systems of Cancer Patients by Ensemble Clustering

Clustering gene expression data based on predicted differential effects of GV interaction.

A Clustering-Based Approach to Predict Outcome in Cancer Patients

Clustering validation by distribution hypothesis learning

Selecting Genes with Dissimilar Discrimination Strength for Sample Class Prediction.

Class-Conditional Conformal Prediction with Many Classes

Clustering with Confidence: Finding Clusters with Statistical Guarantees

Knowing what you know: valid and validated confidence sets in multiclass and multilabel prediction

Efficient Estimation for Rank-Based Regression with Clustered Data

Robust model-based clustering with gene ranking

A Voting Approach to Identify a Small Number of Highly Predictive Genes Using Multiple Classifiers

Sparse clusterability: testing for cluster structure in high dimensions

Normalised clustering accuracy: An asymmetric external cluster validity measure

Statistical Significance of Clustering using Soft Thresholding

Hierarchical clustered multiclass discriminant analysis via cross-validation

Seminal Quality Prediction Using Clustering-Based Decision Forests

Statistical power for cluster analysis

Extension of the Dip-test Repertoire -- Efficient and Differentiable p-value Calculation for Clustering