Abstract:In the measurement-constrained problems, despite the availability of large datasets, we may be only affordable to observe the labels on a small portion of the large dataset. This poses a critical question that which data points are most beneficial to label given a budget constraint. In this paper, we focus on the estimation of the optimal individualized threshold in a measurement-constrained M-estimation framework. Our goal is to estimate a high-dimensional parameter $\theta$ in a linear threshold $\theta^T Z$ for a continuous variable $X$ such that the discrepancy between whether $X$ exceeds the threshold $\theta^T Z$ and a binary outcome $Y$ is minimized. We propose a novel $K$-step active subsampling algorithm to estimate $\theta$, which iteratively samples the most informative observations and solves a regularized M-estimator. The theoretical properties of our estimator demonstrate a phase transition phenomenon with respect to $\beta\geq 1$, the smoothness of the conditional density of $X$ given $Y$ and $Z$. For $\beta>(1+\sqrt{3})/2$, we show that the two-step algorithm yields an estimator with the parametric convergence rate $O_p((s \log d /N)^{1/2})$ in $l_2$ norm. The rate of our estimator is strictly faster than the minimax optimal rate with $N$ i.i.d. samples drawn from the population. For the other two scenarios $1<\beta\leq (1+\sqrt{3})/2$ and $\beta=1$, the estimator from the two-step algorithm is sub-optimal. The former requires to run $K>2$ steps to attain the same parametric rate, whereas in the latter case only a near parametric rate can be obtained. Furthermore, we formulate a minimax framework for the measurement-constrained M-estimation problem and prove that our estimator is minimax rate optimal up to a logarithmic factor. Finally, we demonstrate the performance of our method in simulation studies and apply the method to analyze a large diabetes dataset.

Detecting Rare and Faint Signals Via Thresholding Maximum Likelihood Estimators

A Novel Near Maximum Likelihood Detection Scheme for Wireless Mimo Systems

A two-step strategy for detecting differential gene expression in cDNA microarray data

A Robust Test for Threshold‐Type Nonlinearity in Multivariate Time Series Analysis

Test of Significance for High-dimensional Thresholds with Application to Individualized Minimal Clinically Important Difference

Signal Identification for Rare and Weak Features: Higher Criticism or False Discovery Rates?

Unit-Free and Robust Detection of Differential Expression from RNA-Seq Data

Empirical Likelihood Based Tests for Detecting the Presence of Significant Predictors in Marginal Quantile Regression

Identifying differentially expressed genes in human acute leukemia and mouse brain microarray datasets utilizing QTModel

Determining a Detectable Threshold of Signal Intensity in Cdna Microarray Based on Accumulated Distribution.

A Unified Model for Joint Normalization and Differential Gene Expression Detection in RNA-Seq Data.

Minimax detection boundary and sharp optimal test for Gaussian graphical models

SASeq: A Selective and Adaptive Shrinkage Approach to Detect and Quantify Active Transcripts using RNA-Seq

Weak Signal Identification and Inference in Penalized Likelihood Models for Categorical Responses

A Robust Threshold T Linear Mixed Model for Subgroup Identification Using Multivariate T Distributions

Active Subsampling for Measurement-Constrained M-Estimation of Individualized Thresholds with High-Dimensional Data

ECME Thresholding Methods for Sparse Signal Reconstruction

Threshold Selection and Resource Allocation for Quantized Identification

Simultaneous Detection of Signal Regions Using Quadratic Scan Statistics With Applications in Whole Genome Association Studies

Sequential Detection of Transient Signals with Exponential Family Distribution

Simultaneous Detection of Signal Regions Using Quadratic Scan Statistics With Applications to Whole Genome Association Studies