Abstract:We consider two-class linear classification in a high-dimensional, low-sample size setting. Only a small fraction of the features are useful, the useful features are unknown to us, and each useful feature contributes weakly to the classification decision -- this setting was called the rare/weak model (RW Model). We select features by thresholding feature $z$-scores. The threshold is set by {\it higher criticism} (HC). Let $\pee_i$ denote the $P$-value associated to the $i$-th $z$-score and $\pee_{(i)}$ denote the $i$-th order statistic of the collection of $P$-values. The HC threshold (HCT) is the order statistic of the $z$-score corresponding to index $i$ maximizing $(i/n - \pee_{(i)})/\sqrt{\pee_{(i)}(1-\pee_{(i)})}$. The ideal threshold optimizes the classification error. In \cite{PNAS} we showed that HCT was numerically close to the ideal threshold. We formalize an asymptotic framework for studying the RW model, considering a sequence of problems with increasingly many features and relatively fewer observations. We show that along this sequence, the limiting performance of ideal HCT is essentially just as good as the limiting performance of ideal thresholding. Our results describe two-dimensional {\it phase space}, a two-dimensional diagram with coordinates quantifying "rare" and "weak" in the RW model. Phase space can be partitioned into two regions -- one where ideal threshold classification is successful, and one where the features are so weak and so rare that it must fail. Surprisingly, the regions where ideal HCT succeeds and fails make the exact same partition of the phase diagram. Other threshold methods, such as FDR threshold selection, are successful in a substantially smaller region of the phase space than either HCT or Ideal thresholding.

Signal Identification for Rare and Weak Features: Higher Criticism or False Discovery Rates?

Feature selection by Higher Criticism thresholding: optimal phase diagram

Identification of Signal, Noise, and Indistinguishable Subsets in High-Dimensional Data Analysis

Cellwise Outlier Detection with False Discovery Rate Control

High-Dimensional False Discovery Rate Control for Dependent Variables

Learning false discovery rates by fitting sigmoidal threshold functions

Local False Discovery Rate Estimation with Competition-Based Procedures for Variable Selection

Test of Significance for High-dimensional Thresholds with Application to Individualized Minimal Clinically Important Difference

Detecting Rare and Faint Signals Via Thresholding Maximum Likelihood Estimators

False Discovery Rate Control in Cancer Biomarker Selection Using Knockoffs.

Controlling the False Discovery Rate in Subspace Selection

Thresholding of Statistical Maps in Functional Neuroimaging Using the False Discovery Rate

Weak Signal Inclusion under Dependence and Applications in Genome-wide Association Study

Thresholding the higher criticism test statistics for optimality in a heterogeneous setting

Optimal control of false discovery criteria in the two-group model

Modifying the false discovery rate procedure based on the information theory under arbitrary correlation structure and its performance in high-dimensional genomic data

Detecting rare and weak deviations of non-proportional hazard in survival analysis

False Discovery Rate Control for Confounder Selection Using Mirror Statistics

False discovery rate control: Moving beyond the Benjamini–Hochberg method

Controlling the False Discovery Rate for Binary Feature Selection via Knockoff

Transcriptome data are insufficient to control false discoveries in regulatory network inference