Discussion: The Dantzig selector: Statistical estimation when $p$ is much larger than $n$

Michael P. Friedlander,Michael A. Saunders

DOI: https://doi.org/10.48550/arXiv.0803.3135

2008-03-21

Statistics Theory

Abstract:Discussion of ``The Dantzig selector: Statistical estimation when $p$ is much larger than $n$'' [math/0506081]

What problem does this paper attempt to address?

Rejoinder: The Dantzig selector: Statistical estimation when $p$ is much larger than $n$

Emmanuel Candès,Terence Tao

DOI: https://doi.org/10.48550/arXiv.0803.3136

2008-03-21

Statistics Theory

Abstract:Rejoinder to ``The Dantzig selector: Statistical estimation when $p$ is much larger than $n$'' [math/0506081]
Discussion: A tale of three cousins: Lasso, L2Boosting and Dantzig

N. Meinshausen,G. Rocha,B. Yu

DOI: https://doi.org/10.1214/009053607000000460

2008-03-21

Abstract:Discussion of ``The Dantzig selector: Statistical estimation when $p$ is much larger than $n$'' by Emmanuel Candes and Terence Tao [<a class="link-https" data-arxiv-id="math/0506081" href="https://arxiv.org/abs/math/0506081">math/0506081</a>]

Statistics Theory
Parallelism, uniqueness, and large-sample asymptotics for the Dantzig selector

Lee Dicker,Xihong Lin

DOI: https://doi.org/10.1002/cjs.11151

2013-03-01

Abstract:The Dantzig selector (Candès and Tao, 2007) is a popular ℓ1-regularization method for variable selection and estimation in linear regression. We present a very weak geometric condition on the observed predictors which is related to parallelism and, when satisfied, ensures the uniqueness of Dantzig selector estimators. The condition holds with probability 1, if the predictors are drawn from a continuous distribution. We discuss the necessity of this condition for uniqueness and also provide a closely related condition which ensures uniqueness of lasso estimators (Tibshirani, 1996). Large sample asymptotics for the Dantzig selector, i.e. almost sure convergence and the asymptotic distribution, follow directly from our uniqueness results and a continuity argument. The limiting distribution of the Dantzig selector is generally non-normal. Though our asymptotic results require that the number of predictors is fixed (similar to (Knight and Fu, 2000)), our uniqueness results are valid for an arbitrary number of predictors and observations.
Two step estimations via the Dantzig selector for models of stochastic processes with high-dimensional parameters

Kou Fujimori,Koji Tsukuda

2024-09-18

Abstract:We consider the sparse estimation for stochastic processes with possibly infinite-dimensional nuisance parameters, by using the Dantzig selector which is a sparse estimation method similar to $Z$-estimation. When a consistent estimator for a nuisance parameter is obtained, it is possible to construct an asymptotically normal estimator for the parameter of interest under appropriate conditions. Motivated by this fact, we establish the asymptotic behavior of the Dantzig selector for models of ergodic stochastic processes with high-dimensional parameters of interest and possibly infinite-dimensional nuisance parameters. Moreover, we construct an asymptotically normal estimator by the two step estimation with help of the variable selection through the Dantzig selector and a consistent estimator of the nuisance parameter. Applications to ergodic time series models including integer-valued autoregressive models and ergodic diffusion processes are presented.

Statistics Theory
The constrained Dantzig selector with enhanced consistency

Yinfei Kong,Zemin Zheng,Jinchi Lv

DOI: https://doi.org/10.48550/arXiv.1605.03311

2016-05-11

Abstract:The Dantzig selector has received popularity for many applications such as compressed sensing and sparse modeling, thanks to its computational efficiency as a linear programming problem and its nice sampling properties. Existing results show that it can recover sparse signals mimicking the accuracy of the ideal procedure, up to a logarithmic factor of the dimensionality. Such a factor has been shown to hold for many regularization methods. An important question is whether this factor can be reduced to a logarithmic factor of the sample size in ultra-high dimensions under mild regularity conditions. To provide an affirmative answer, in this paper we suggest the constrained Dantzig selector, which has more flexible constraints and parameter space. We prove that the suggested method can achieve convergence rates within a logarithmic factor of the sample size of the oracle rates and improved sparsity, under a fairly weak assumption on the signal strength. Such improvement is significant in ultra-high dimensions. This method can be implemented efficiently through sequential linear programming. Numerical studies confirm that the sample size needed for a certain level of accuracy in these problems can be much reduced.

Methodology,Machine Learning
Simultaneous analysis of Lasso and Dantzig selector

Peter J. Bickel,Ya'acov Ritov,Alexandre B. Tsybakov

DOI: https://doi.org/10.1214/08-AOS620

2010-11-09

Abstract:We exhibit an approximate equivalence between the Lasso estimator and Dantzig selector. For both methods we derive parallel oracle inequalities for the prediction risk in the general nonparametric regression model, as well as bounds on the $\ell_p$ estimation loss for $1\le p\le 2$ in the linear model when the number of variables can be much larger than the sample size.

Statistics Theory
The Dantzig selector for a linear model of diffusion processes

Kou Fujimori

DOI: https://doi.org/10.48550/arXiv.1709.00710

2017-09-03

Abstract:In this paper, a linear model of diffusion processes with unknown drift and diagonal diffusion matrices is discussed. We will consider the estimation problems for unknown parameters based on the discrete time observation in high-dimensional and sparse settings. To estimate drift matrices, the Dantzig selector which was proposed by Candés and Tao in 2007 will be applied. Then, we will prove two types of consistency of the estimator of drift matrix; one is the consistency in the sense of $l_q$ norm for every $q \in [1,\infty]$ and the other is the variable selection consistency. Moreover, we will construct an asymptotically normal estimator of the drift matrix by using the variable selection consistency of the Dantzig selector.

Statistics Theory
The Dantzig selector for diffusion processes with covariates

Kou Fujimori,Yoichi Nishiyama

DOI: https://doi.org/10.48550/arXiv.1611.10011

2016-11-30

Abstract:The Dantzig selector for a special parametric model of diffusion processes is studied in this paper. In our model, the diffusion coefficient is given as the exponential of the linear combination of other processes which are regarded as covariates. We propose an estimation procedure which is an adaptation of the Dantzig selector for linear regression models and prove the $l_q$ consistency of the estimator for all $q \in [1,\infty]$.

Statistics Theory
The Linearized Alternating Direction Method of Multipliers for Dantzig Selector

Xiangfeng Wang,Xiaoming Yuan

DOI: https://doi.org/10.1137/110833543

IF: 2.968

2012-01-01

SIAM Journal on Scientific Computing

Abstract:The Dantzig selector was recently proposed to perform variable selection and model fitting in the linear regression model. It can be solved numerically by the alternating direction method of multipliers (ADM); and in this paper, we show that the application of ADM to the Dantzig selector can be speeded up significantly if one of its resulting subproblems at each iteration is linearized. The resulting linearized ADM for the Dantzig selector is shown to be efficient for solving both synthetic and real world data sets.

mathematics, applied
High-dimensional stochastic optimization with the generalized Dantzig estimator

Karim Lounici

DOI: https://doi.org/10.48550/arXiv.0811.2281

2008-11-14

Abstract:We propose a generalized version of the Dantzig selector. We show that it satisfies sparsity oracle inequalities in prediction and estimation. We consider then the particular case of high-dimensional linear regression model selection with the Huber loss function. In this case we derive the sup-norm convergence rate and the sign concentration property of the Dantzig estimators under a mutual coherence assumption on the dictionary.

Statistics Theory
Computing Estimators of Dantzig Selector type via Column and Constraint Generation

Rahul Mazumder,Stephen Wright,Andrew Zheng

DOI: https://doi.org/10.48550/arXiv.1908.06515

2019-08-19

Abstract:We consider a class of linear-programming based estimators in reconstructing a sparse signal from linear measurements. Specific formulations of the reconstruction problem considered here include Dantzig selector, basis pursuit (for the case in which the measurements contain no errors), and the fused Dantzig selector (for the case in which the underlying signal is piecewise constant). In spite of being estimators central to sparse signal processing and machine learning, solving these linear programming problems for large scale instances remains a challenging task, thereby limiting their usage in practice. We show that classic constraint- and column-generation techniques from large scale linear programming, when used in conjunction with a commercial implementation of the simplex method, and initialized with the solution from a closely-related Lasso formulation, yields solutions with high efficiency in many settings.

Computation,Optimization and Control,Machine Learning
Selective inference is easier with p-values

Anav Sood

2024-11-21

Abstract:Selective inference is a subfield of statistics that enables valid inference after selection of a data-dependent question. In this paper, we introduce selectively dominant p-values, a class of p-values that allow practitioners to easily perform inference after arbitrary selection procedures. Unlike a traditional p-value, whose distribution must stochastically dominate the uniform distribution under the null, a selectively dominant p-value must have a post-selection distribution that stochastically dominates that of a uniform having undergone the same selection process; moreover, this property must hold simultaneously for all possible selection processes. Despite the strength of this condition, we show that all commonly used p-values (e.g., p-values from two-sided testing in parametric families, one-sided testing in monotone likelihood ratio and exponential families, $F$-tests for linear regression, and permutation tests) are selectively dominant. By recasting two canonical selective inference problems-inference on winners and rank verification-in our selective dominance framework, we provide simpler derivations, a deeper conceptual understanding, and new generalizations and variations of these methods. Additionally, we use our insights to introduce selective variants of methods that combine p-values, such as Fisher's combination test.

Methodology,Statistics Theory
Generalized Dantzig Selector: Application to the k-support norm

Soumyadeep Chatterjee,Sheng Chen,Arindam Banerjee

DOI: https://doi.org/10.48550/arXiv.1406.5291

2015-02-03

Abstract:We propose a Generalized Dantzig Selector (GDS) for linear models, in which any norm encoding the parameter structure can be leveraged for estimation. We investigate both computational and statistical aspects of the GDS. Based on conjugate proximal operator, a flexible inexact ADMM framework is designed for solving GDS, and non-asymptotic high-probability bounds are established on the estimation error, which rely on Gaussian width of unit norm ball and suitable set encompassing estimation error. Further, we consider a non-trivial example of the GDS using $k$-support norm. We derive an efficient method to compute the proximal operator for $k$-support norm since existing methods are inapplicable in this setting. For statistical analysis, we provide upper bounds for the Gaussian widths needed in the GDS analysis, yielding the first statistical recovery guarantee for estimation with the $k$-support norm. The experimental results confirm our theoretical analysis.

Machine Learning
An Alternating Direction Method for Finding Dantzig Selectors

Zhaosong Lu,Ting Kei Pong,Yong Zhang

DOI: https://doi.org/10.48550/arXiv.1011.4604

2010-11-21

Abstract:In this paper, we study the alternating direction method for finding the Dantzig selectors, which are first introduced in [8]. In particular, at each iteration we apply the nonmonotone gradient method proposed in [17] to approximately solve one subproblem of this method. We compare our approach with a first-order method proposed in [3]. The computational results show that our approach usually outperforms that method in terms of CPU time while producing solutions of comparable quality.

Computation
Precise Asymptotics in the Baum–Katz and Davis Laws of Large Numbers of P-Mixing Sequences

Wei Huang,Ye Jiang,Li Xin Zhang

DOI: https://doi.org/10.1007/s10114-004-0452-x

2005-01-01

Abstract:Let X , X n ; n ≥ 1 be a strictly stationary sequence of p -mixing random variables with mean zeros and finite variances. Set S_n = ∑_k = 1^n X_k , M_n = max _k ⩽ n| S_k|, n ⩾ 1. Suppose lim _n →∞ES^2_n /n = :σ ^2 > 0 and ∑_n = 1^∞ρ ^2/d (2^n ) < ∞ , where d = 2 if 1 ≤ r < 2 and d > r if r ≥ 2. We prove that if E∣ X ∣ r < ∞, for 1 ≤ p < 2 and r > p , then lim_ε↘∞ε ^2(r - p)/2 - p∑_n = 1^∞n^r/p - 2P{M_n⩾ε n^1/p} = 2p/r - p∑_k = 0^∞( - 1)^k/(2k + 1)^2(r - p)/(2 - p)E| Z |^2(r - p)/2 - p, where Z has a normal distribution with mean 0 and variance σ 2 .
The Discrete Dantzig Selector: Estimating Sparse Linear Models via Mixed Integer Linear Optimization

Rahul Mazumder,Peter Radchenko

DOI: https://doi.org/10.48550/arXiv.1508.01922

2017-01-19

Abstract:We propose a novel high-dimensional linear regression estimator: the Discrete Dantzig Selector, which minimizes the number of nonzero regression coefficients subject to a budget on the maximal absolute correlation between the features and residuals. Motivated by the significant advances in integer optimization over the past 10-15 years, we present a Mixed Integer Linear Optimization (MILO) approach to obtain certifiably optimal global solutions to this nonconvex optimization problem. The current state of algorithmics in integer optimization makes our proposal substantially more computationally attractive than the least squares subset selection framework based on integer quadratic optimization, recently proposed in [8] and the continuous nonconvex quadratic optimization framework of [33]. We propose new discrete first-order methods, which when paired with state-of-the-art MILO solvers, lead to good solutions for the Discrete Dantzig Selector problem for a given computational budget. We illustrate that our integrated approach provides globally optimal solutions in significantly shorter computation times, when compared to off-the-shelf MILO solvers. We demonstrate both theoretically and empirically that in a wide range of regimes the statistical properties of the Discrete Dantzig Selector are superior to those of popular $\ell_{1}$-based approaches. We illustrate that our approach can handle problem instances with p = 10,000 features with certifiable optimality making it a highly scalable combinatorial variable selection approach in sparse linear modeling.

Methodology,Optimization and Control,Statistics Theory,Computation,Machine Learning
A partially proximal linearized alternating minimization method for finding Dantzig selectors

Xiaoyu Mao,Hongjin He,Hong-Kun Xu

DOI: https://doi.org/10.1007/s40314-021-01450-5

2021-02-26

Computational and Applied Mathematics

Abstract:The Dantzig selector (DS) is an efficient estimator designed for high-dimensional linear regression problems, especially for the case where the number of samples n is much less than the dimension of features (or variables) p. In this paper, we first reformulate the underlying DS model as an unconstrained minimization problem of the sum of two nonsmooth convex functions and a smooth coupled function. Then by exploiting the structure of the resulting model, we propose a partially proximal linearized alternating minimization method (P-PLAM), whose two subproblems are easy enough with closed-form solutions. Another remarkable advantage is that P-PLAM requires only one starting point, which is potentially helpful for saving computing time. A series of computational experiments on synthetic and real-world data sets demonstrate that the proposed P-PLAM has promising numerical performance in the sense that P-PLAM can obtain higher quality solutions by taking less computing time than some existing state-of-the-art first-order solvers.

mathematics, applied
Opportunistic Detection under a Fixed-Sample-Size Setting

Wenyi Zhang,H. Vincent Poor

DOI: https://doi.org/10.1109/tit.2012.2226204

IF: 2.5

2012-01-01

IEEE Transactions on Information Theory

Abstract:With a finite number of samples drawn from one of two possible distributions sequentially revealed, an opportunistic detection rule is proposed, which possibly makes an early decision in favor of the alternative hypothesis, while always deferring the decision of the null hypothesis until collecting all the samples. Properties of this opportunistic detection rule are discussed and its key asymptotic behavior in the large sample size limit is established. Specifically, a Chernoff-Stein lemma type of characterization of the exponential decay rate of the miss probability under the Neyman-Pearson criterion is established, and consequently, a performance metric of asymptotic exponential efficiency loss is proposed and discussed, which is exactly the ratio between the Kullback-Leibler distance and the Chernoff information of the two hypotheses. Analytical results are corroborated by numerical experiments.
Testing Support Size More Efficiently Than Learning Histograms

Renato Ferreira Pinto Jr.,Nathaniel Harms

2024-10-25

Abstract:Consider two problems about an unknown probability distribution $p$: 1. How many samples from $p$ are required to test if $p$ is supported on $n$ elements or not? Specifically, given samples from $p$, determine whether it is supported on at most $n$ elements, or it is "$\epsilon$-far" (in total variation distance) from being supported on $n$ elements. 2. Given $m$ samples from $p$, what is the largest lower bound on its support size that we can produce? The best known upper bound for problem (1) uses a general algorithm for learning the histogram of the distribution $p$, which requires $\Theta(\tfrac{n}{\epsilon^2 \log n})$ samples. We show that testing can be done more efficiently than learning the histogram, using only $O(\tfrac{n}{\epsilon \log n} \log(1/\epsilon))$ samples, nearly matching the best known lower bound of $\Omega(\tfrac{n}{\epsilon \log n})$. This algorithm also provides a better solution to problem (2), producing larger lower bounds on support size than what follows from previous work. The proof relies on an analysis of Chebyshev polynomial approximations outside the range where they are designed to be good approximations, and the paper is intended as an accessible self-contained exposition of the Chebyshev polynomial method.

Data Structures and Algorithms,Machine Learning
The Price of Selection in Differential Privacy

Mitali Bafna,Jonathan Ullman

DOI: https://doi.org/10.48550/arXiv.1702.02970

2017-02-10

Abstract:In the differentially private top-$k$ selection problem, we are given a dataset $X \in \{\pm 1\}^{n \times d}$, in which each row belongs to an individual and each column corresponds to some binary attribute, and our goal is to find a set of $k \ll d$ columns whose means are approximately as large as possible. Differential privacy requires that our choice of these $k$ columns does not depend too much on any on individual's dataset. This problem can be solved using the well known exponential mechanism and composition properties of differential privacy. In the high-accuracy regime, where we require the error of the selection procedure to be to be smaller than the so-called sampling error $\alpha \approx \sqrt{\ln(d)/n}$, this procedure succeeds given a dataset of size $n \gtrsim k \ln(d)$. We prove a matching lower bound, showing that a dataset of size $n \gtrsim k \ln(d)$ is necessary for private top-$k$ selection in this high-accuracy regime. Our lower bound is the first to show that selecting the $k$ largest columns requires more data than simply estimating the value of those $k$ columns, which can be done using a dataset of size just $n \gtrsim k$.

Data Structures and Algorithms,Cryptography and Security

Discussion: The Dantzig selector: Statistical estimation when $p$ is much larger than $n$

Rejoinder: The Dantzig selector: Statistical estimation when $p$ is much larger than $n$

Discussion: A tale of three cousins: Lasso, L2Boosting and Dantzig

Parallelism, uniqueness, and large-sample asymptotics for the Dantzig selector

Two step estimations via the Dantzig selector for models of stochastic processes with high-dimensional parameters

The constrained Dantzig selector with enhanced consistency

Simultaneous analysis of Lasso and Dantzig selector

The Dantzig selector for a linear model of diffusion processes

The Dantzig selector for diffusion processes with covariates

The Linearized Alternating Direction Method of Multipliers for Dantzig Selector

High-dimensional stochastic optimization with the generalized Dantzig estimator

Computing Estimators of Dantzig Selector type via Column and Constraint Generation

Selective inference is easier with p-values

Generalized Dantzig Selector: Application to the k-support norm

An Alternating Direction Method for Finding Dantzig Selectors

Precise Asymptotics in the Baum–Katz and Davis Laws of Large Numbers of P-Mixing Sequences

The Discrete Dantzig Selector: Estimating Sparse Linear Models via Mixed Integer Linear Optimization

A partially proximal linearized alternating minimization method for finding Dantzig selectors

Opportunistic Detection under a Fixed-Sample-Size Setting

Testing Support Size More Efficiently Than Learning Histograms

The Price of Selection in Differential Privacy