Rejoinder: The Dantzig selector: Statistical estimation when $p$ is much larger than $n$

Emmanuel Candès,Terence Tao

DOI: https://doi.org/10.48550/arXiv.0803.3136

2008-03-21

Statistics Theory

Abstract:Rejoinder to ``The Dantzig selector: Statistical estimation when $p$ is much larger than $n$'' [math/0506081]

What problem does this paper attempt to address?

Discussion: The Dantzig selector: Statistical estimation when $p$ is much larger than $n$

Michael P. Friedlander,Michael A. Saunders

DOI: https://doi.org/10.48550/arXiv.0803.3135

2008-03-21

Statistics Theory

Abstract:Discussion of ``The Dantzig selector: Statistical estimation when $p$ is much larger than $n$'' [math/0506081]
Discussion: A tale of three cousins: Lasso, L2Boosting and Dantzig

N. Meinshausen,G. Rocha,B. Yu

DOI: https://doi.org/10.1214/009053607000000460

2008-03-21

Abstract:Discussion of ``The Dantzig selector: Statistical estimation when $p$ is much larger than $n$'' by Emmanuel Candes and Terence Tao [<a class="link-https" data-arxiv-id="math/0506081" href="https://arxiv.org/abs/math/0506081">math/0506081</a>]

Statistics Theory
Parallelism, uniqueness, and large-sample asymptotics for the Dantzig selector

Lee Dicker,Xihong Lin

DOI: https://doi.org/10.1002/cjs.11151

2013-03-01

Abstract:The Dantzig selector (Candès and Tao, 2007) is a popular ℓ1-regularization method for variable selection and estimation in linear regression. We present a very weak geometric condition on the observed predictors which is related to parallelism and, when satisfied, ensures the uniqueness of Dantzig selector estimators. The condition holds with probability 1, if the predictors are drawn from a continuous distribution. We discuss the necessity of this condition for uniqueness and also provide a closely related condition which ensures uniqueness of lasso estimators (Tibshirani, 1996). Large sample asymptotics for the Dantzig selector, i.e. almost sure convergence and the asymptotic distribution, follow directly from our uniqueness results and a continuity argument. The limiting distribution of the Dantzig selector is generally non-normal. Though our asymptotic results require that the number of predictors is fixed (similar to (Knight and Fu, 2000)), our uniqueness results are valid for an arbitrary number of predictors and observations.
Simultaneous analysis of Lasso and Dantzig selector

Peter J. Bickel,Ya'acov Ritov,Alexandre B. Tsybakov

DOI: https://doi.org/10.1214/08-AOS620

2010-11-09

Abstract:We exhibit an approximate equivalence between the Lasso estimator and Dantzig selector. For both methods we derive parallel oracle inequalities for the prediction risk in the general nonparametric regression model, as well as bounds on the $\ell_p$ estimation loss for $1\le p\le 2$ in the linear model when the number of variables can be much larger than the sample size.

Statistics Theory
Two step estimations via the Dantzig selector for models of stochastic processes with high-dimensional parameters

Kou Fujimori,Koji Tsukuda

2024-09-18

Abstract:We consider the sparse estimation for stochastic processes with possibly infinite-dimensional nuisance parameters, by using the Dantzig selector which is a sparse estimation method similar to $Z$-estimation. When a consistent estimator for a nuisance parameter is obtained, it is possible to construct an asymptotically normal estimator for the parameter of interest under appropriate conditions. Motivated by this fact, we establish the asymptotic behavior of the Dantzig selector for models of ergodic stochastic processes with high-dimensional parameters of interest and possibly infinite-dimensional nuisance parameters. Moreover, we construct an asymptotically normal estimator by the two step estimation with help of the variable selection through the Dantzig selector and a consistent estimator of the nuisance parameter. Applications to ergodic time series models including integer-valued autoregressive models and ergodic diffusion processes are presented.

Statistics Theory
The constrained Dantzig selector with enhanced consistency

Yinfei Kong,Zemin Zheng,Jinchi Lv

DOI: https://doi.org/10.48550/arXiv.1605.03311

2016-05-11

Abstract:The Dantzig selector has received popularity for many applications such as compressed sensing and sparse modeling, thanks to its computational efficiency as a linear programming problem and its nice sampling properties. Existing results show that it can recover sparse signals mimicking the accuracy of the ideal procedure, up to a logarithmic factor of the dimensionality. Such a factor has been shown to hold for many regularization methods. An important question is whether this factor can be reduced to a logarithmic factor of the sample size in ultra-high dimensions under mild regularity conditions. To provide an affirmative answer, in this paper we suggest the constrained Dantzig selector, which has more flexible constraints and parameter space. We prove that the suggested method can achieve convergence rates within a logarithmic factor of the sample size of the oracle rates and improved sparsity, under a fairly weak assumption on the signal strength. Such improvement is significant in ultra-high dimensions. This method can be implemented efficiently through sequential linear programming. Numerical studies confirm that the sample size needed for a certain level of accuracy in these problems can be much reduced.

Methodology,Machine Learning
The Dantzig selector for a linear model of diffusion processes

Kou Fujimori

DOI: https://doi.org/10.48550/arXiv.1709.00710

2017-09-03

Abstract:In this paper, a linear model of diffusion processes with unknown drift and diagonal diffusion matrices is discussed. We will consider the estimation problems for unknown parameters based on the discrete time observation in high-dimensional and sparse settings. To estimate drift matrices, the Dantzig selector which was proposed by Candés and Tao in 2007 will be applied. Then, we will prove two types of consistency of the estimator of drift matrix; one is the consistency in the sense of $l_q$ norm for every $q \in [1,\infty]$ and the other is the variable selection consistency. Moreover, we will construct an asymptotically normal estimator of the drift matrix by using the variable selection consistency of the Dantzig selector.

Statistics Theory
The Dantzig selector for diffusion processes with covariates

Kou Fujimori,Yoichi Nishiyama

DOI: https://doi.org/10.48550/arXiv.1611.10011

2016-11-30

Abstract:The Dantzig selector for a special parametric model of diffusion processes is studied in this paper. In our model, the diffusion coefficient is given as the exponential of the linear combination of other processes which are regarded as covariates. We propose an estimation procedure which is an adaptation of the Dantzig selector for linear regression models and prove the $l_q$ consistency of the estimator for all $q \in [1,\infty]$.

Statistics Theory
Selective inference is easier with p-values

Anav Sood

2024-11-21

Abstract:Selective inference is a subfield of statistics that enables valid inference after selection of a data-dependent question. In this paper, we introduce selectively dominant p-values, a class of p-values that allow practitioners to easily perform inference after arbitrary selection procedures. Unlike a traditional p-value, whose distribution must stochastically dominate the uniform distribution under the null, a selectively dominant p-value must have a post-selection distribution that stochastically dominates that of a uniform having undergone the same selection process; moreover, this property must hold simultaneously for all possible selection processes. Despite the strength of this condition, we show that all commonly used p-values (e.g., p-values from two-sided testing in parametric families, one-sided testing in monotone likelihood ratio and exponential families, $F$-tests for linear regression, and permutation tests) are selectively dominant. By recasting two canonical selective inference problems-inference on winners and rank verification-in our selective dominance framework, we provide simpler derivations, a deeper conceptual understanding, and new generalizations and variations of these methods. Additionally, we use our insights to introduce selective variants of methods that combine p-values, such as Fisher's combination test.

Methodology,Statistics Theory
High-dimensional stochastic optimization with the generalized Dantzig estimator

Karim Lounici

DOI: https://doi.org/10.48550/arXiv.0811.2281

2008-11-14

Abstract:We propose a generalized version of the Dantzig selector. We show that it satisfies sparsity oracle inequalities in prediction and estimation. We consider then the particular case of high-dimensional linear regression model selection with the Huber loss function. In this case we derive the sup-norm convergence rate and the sign concentration property of the Dantzig estimators under a mutual coherence assumption on the dictionary.

Statistics Theory
Sorting and Selection in Rounds with Adversarial Comparisons

Christopher Trevisan

2023-10-14

Abstract:We continue the study of selection and sorting of $n$ numbers under the adversarial comparator model, where comparisons can be adversarially tampered with if the arguments are sufficiently close. We derive a randomized sorting algorithm that does $O(n \log^2 n)$ comparisons and gives a correct answer with high probability, addressing an open problem of Ajtai, Feldman, Hassadim, and Nelson [AFHN15]. Our algorithm also implies a selection algorithm that does $O(n \log n)$ comparisons and gives a correct answer with high probability. Both of these results are a $\log$ factor away from the naive lower bound. [AFHN15] shows an $\Omega(n^{1+\varepsilon})$ lower bound for both sorting and selection in the deterministic case, so our results also prove a discrepancy between what is possible with deterministic and randomized algorithms in this setting. We also consider both sorting and selection in rounds, exploring the tradeoff between accuracy, number of comparisons, and number of rounds. Using results from sorting networks, we give general algorithms for sorting in $d$ rounds where the number of comparisons increases with $d$ and the accuracy decreases with $d$. Using these algorithms, we derive selection algorithms in $d+O(\log d)$ rounds that use the same number of comparisons as the corresponding sorting algorithm, but have a constant accuracy. Notably, this gives selection algorithms in $d$ rounds that use $n^{1 + o(1)}$ comparisons and have constant accuracy for all $d = \omega(1)$, which still beats the deterministic lower bound of $\Omega(n^{1+\varepsilon})$.

Data Structures and Algorithms
Precise Asymptotics in the Baum–Katz and Davis Laws of Large Numbers of P-Mixing Sequences

Wei Huang,Ye Jiang,Li Xin Zhang

DOI: https://doi.org/10.1007/s10114-004-0452-x

2005-01-01

Abstract:Let X , X n ; n ≥ 1 be a strictly stationary sequence of p -mixing random variables with mean zeros and finite variances. Set S_n = ∑_k = 1^n X_k , M_n = max _k ⩽ n| S_k|, n ⩾ 1. Suppose lim _n →∞ES^2_n /n = :σ ^2 > 0 and ∑_n = 1^∞ρ ^2/d (2^n ) < ∞ , where d = 2 if 1 ≤ r < 2 and d > r if r ≥ 2. We prove that if E∣ X ∣ r < ∞, for 1 ≤ p < 2 and r > p , then lim_ε↘∞ε ^2(r - p)/2 - p∑_n = 1^∞n^r/p - 2P{M_n⩾ε n^1/p} = 2p/r - p∑_k = 0^∞( - 1)^k/(2k + 1)^2(r - p)/(2 - p)E| Z |^2(r - p)/2 - p, where Z has a normal distribution with mean 0 and variance σ 2 .
Selector form of Weaver's conjecture, Feichtinger's conjecture, and frame sparsification

Marcin Bownik

2024-05-28

Abstract:We show an extension of a probabilistic result of Marcus, Spielman, and Srivastava, which resolved the Kadison-Singer problem, for block diagonal positive semidefinite random matrices. We use this result to show several selector results, which generalize their partition counterparts. This includes a selector form of Weaver's KS$_r$ conjecture for block diagonal trace class operators, which extends a selector result for Bessel sequences, or equivalently rank one matrices, due to Londner and the author. We also show a selector variant of Feichtinger's conjecture for a (possibly infinite) collection of Bessel sequences, extending earlier results for a single Bessel sequence. We prove a generalization of the $R_\epsilon$ conjecture of Casazza, Tremain, and Vershynin for infinite collection of equal norm Bessel sequences. In particular, our selector result yields a conjectured asymptotically optimal bound for a single Bessel sequence in terms of Riesz sequence tightness parameter. We establish an iterated selector form of Weaver's KS$_2$ conjecture and show its applications. This includes a solution of an open problem on nearly unit norm Parseval frames of exponentials, which was posed by Londner and the author. We generalize a discretization result for continuous frames by Freeman and Speegle in two ways. First, we extend their result from the setting of rank one operators to positive trace operator valued measures. Second, we establish a nearly tight discretization of bounded continuous Parseval frames. In particular, our selector result yields an improvement of the result of Nitzan, Olevskii, and Ulanovskii and implies the existence of nearly tight exponential frames for unbounded sets with an explicit control on their frame redundancy.

Classical Analysis and ODEs,Functional Analysis
Computing Estimators of Dantzig Selector type via Column and Constraint Generation

Rahul Mazumder,Stephen Wright,Andrew Zheng

DOI: https://doi.org/10.48550/arXiv.1908.06515

2019-08-19

Abstract:We consider a class of linear-programming based estimators in reconstructing a sparse signal from linear measurements. Specific formulations of the reconstruction problem considered here include Dantzig selector, basis pursuit (for the case in which the measurements contain no errors), and the fused Dantzig selector (for the case in which the underlying signal is piecewise constant). In spite of being estimators central to sparse signal processing and machine learning, solving these linear programming problems for large scale instances remains a challenging task, thereby limiting their usage in practice. We show that classic constraint- and column-generation techniques from large scale linear programming, when used in conjunction with a commercial implementation of the simplex method, and initialized with the solution from a closely-related Lasso formulation, yields solutions with high efficiency in many settings.

Computation,Optimization and Control,Machine Learning
The Dantzig selector: Recovery of Signal via $\ell_1-α\ell_2$ Minimization

Huanmin Ge,Peng Li

DOI: https://doi.org/10.1088/1361-6420/ac39f8

2021-05-29

Abstract:In the paper, we proposed the Dantzig selector based on the $\ell_{1}-\alpha \ell_{2}$~$(0< \alpha \leq1)$ minimization for the signal recovery. In the Dantzig selector, the constraint $\|{\bf A}^{\top}({\bf b}-{\bf A}{\bf x})\|_\infty \leq \eta$ for some small constant $\eta>0$ means the columns of ${\bf A}$ has very weakly correlated with the error vector ${\bf e}={\bf A}{\bf x}-{\bf b}$. First, recovery guarantees based on the restricted isometry property (RIP) are established for signals. Next, we propose the effective algorithm to solve the proposed Dantzig selector. Last, we illustrate the proposed model and algorithm by extensive numerical experiments for the recovery of signals in the cases of Gaussian, impulsive and uniform noise. And the performance of the proposed Dantzig selector is better than that of the existing methods.

Information Theory,Statistics Theory
The Linearized Alternating Direction Method of Multipliers for Dantzig Selector

Xiangfeng Wang,Xiaoming Yuan

DOI: https://doi.org/10.1137/110833543

IF: 2.968

2012-01-01

SIAM Journal on Scientific Computing

Abstract:The Dantzig selector was recently proposed to perform variable selection and model fitting in the linear regression model. It can be solved numerically by the alternating direction method of multipliers (ADM); and in this paper, we show that the application of ADM to the Dantzig selector can be speeded up significantly if one of its resulting subproblems at each iteration is linearized. The resulting linearized ADM for the Dantzig selector is shown to be efficient for solving both synthetic and real world data sets.

mathematics, applied
An Alternating Direction Method for Finding Dantzig Selectors

Zhaosong Lu,Ting Kei Pong,Yong Zhang

DOI: https://doi.org/10.48550/arXiv.1011.4604

2010-11-21

Abstract:In this paper, we study the alternating direction method for finding the Dantzig selectors, which are first introduced in [8]. In particular, at each iteration we apply the nonmonotone gradient method proposed in [17] to approximately solve one subproblem of this method. We compare our approach with a first-order method proposed in [3]. The computational results show that our approach usually outperforms that method in terms of CPU time while producing solutions of comparable quality.

Computation
Sorting and Selection with Imprecise Comparisons

Miklós Ajtai,Vitaly Feldman,Avinatan Hassidim,Jelani Nelson

DOI: https://doi.org/10.1145/2701427

IF: 1.113

2016-02-12

ACM Transactions on Algorithms

Abstract:We consider a simple model of imprecise comparisons: there exists some δ > 0 such that when a subject is given two elements to compare, if the values of those elements (as perceived by the subject) differ by at least δ, then the comparison will be made correctly; when the two elements have values that are within δ, the outcome of the comparison is unpredictable. This model is inspired by both imprecision in human judgment of values and also by bounded but potentially adversarial errors in the outcomes of sporting tournaments. Our model is closely related to a number of models commonly considered in the psychophysics literature where δ corresponds to the Just Noticeable Difference (JND) unit or difference threshold . In experimental psychology, the method of paired comparisons was proposed as a means for ranking preferences among n elements of a human subject. The method requires performing all ( n 2 ) comparisons, then sorting elements according to the number of wins. The large number of comparisons is performed to counter the potentially faulty decision-making of the human subject, who acts as an imprecise comparator. We show that in our model the method of paired comparisons has optimal accuracy, minimizing the errors introduced by the imprecise comparisons. However, it is also wasteful because it requires all ( n 2 ). We show that the same optimal guarantees can be achieved using 4 n 3/2 comparisons, and we prove the optimality of our method. We then explore the general tradeoff between the guarantees on the error that can be made and number of comparisons for the problems of sorting, max-finding, and selection. Our results provide strong lower bounds and close-to-optimal solutions for each of these problems.

computer science, theory & methods,mathematics, applied
Subadditivity and optimal matching of unbounded samples

Emanuele Caglioti,Michael Goldman,Francesca Pieroni,Dario Trevisan

2024-07-09

Abstract:We obtain new bounds for the optimal matching cost for empirical measures with unbounded support. For a large class of radially symmetric and rapidly decaying probability laws, we prove for the first time the asymptotic rate of convergence for the whole range of power exponents $p$ and dimensions $d$. Moreover we identify the exact prefactor when $p\le d$. We cover in particular the Gaussian case, going far beyond the currently known bounds. Our proof technique is based on approximate sub- and super-additivity bounds along a geometric decomposition adapted to some features the density, such as its radial symmetry and its decay at infinity.

Probability,Mathematical Physics,Functional Analysis,Statistics Theory
The Discrete Dantzig Selector: Estimating Sparse Linear Models via Mixed Integer Linear Optimization

Rahul Mazumder,Peter Radchenko

DOI: https://doi.org/10.48550/arXiv.1508.01922

2017-01-19

Abstract:We propose a novel high-dimensional linear regression estimator: the Discrete Dantzig Selector, which minimizes the number of nonzero regression coefficients subject to a budget on the maximal absolute correlation between the features and residuals. Motivated by the significant advances in integer optimization over the past 10-15 years, we present a Mixed Integer Linear Optimization (MILO) approach to obtain certifiably optimal global solutions to this nonconvex optimization problem. The current state of algorithmics in integer optimization makes our proposal substantially more computationally attractive than the least squares subset selection framework based on integer quadratic optimization, recently proposed in [8] and the continuous nonconvex quadratic optimization framework of [33]. We propose new discrete first-order methods, which when paired with state-of-the-art MILO solvers, lead to good solutions for the Discrete Dantzig Selector problem for a given computational budget. We illustrate that our integrated approach provides globally optimal solutions in significantly shorter computation times, when compared to off-the-shelf MILO solvers. We demonstrate both theoretically and empirically that in a wide range of regimes the statistical properties of the Discrete Dantzig Selector are superior to those of popular $\ell_{1}$-based approaches. We illustrate that our approach can handle problem instances with p = 10,000 features with certifiable optimality making it a highly scalable combinatorial variable selection approach in sparse linear modeling.

Methodology,Optimization and Control,Statistics Theory,Computation,Machine Learning

Rejoinder: The Dantzig selector: Statistical estimation when $p$ is much larger than $n$

Discussion: The Dantzig selector: Statistical estimation when $p$ is much larger than $n$

Discussion: A tale of three cousins: Lasso, L2Boosting and Dantzig

Parallelism, uniqueness, and large-sample asymptotics for the Dantzig selector

Simultaneous analysis of Lasso and Dantzig selector

Two step estimations via the Dantzig selector for models of stochastic processes with high-dimensional parameters

The constrained Dantzig selector with enhanced consistency

The Dantzig selector for a linear model of diffusion processes

The Dantzig selector for diffusion processes with covariates

Selective inference is easier with p-values

High-dimensional stochastic optimization with the generalized Dantzig estimator

Sorting and Selection in Rounds with Adversarial Comparisons

Precise Asymptotics in the Baum–Katz and Davis Laws of Large Numbers of P-Mixing Sequences

Selector form of Weaver's conjecture, Feichtinger's conjecture, and frame sparsification

Computing Estimators of Dantzig Selector type via Column and Constraint Generation

The Dantzig selector: Recovery of Signal via $\ell_1-α\ell_2$ Minimization

The Linearized Alternating Direction Method of Multipliers for Dantzig Selector

An Alternating Direction Method for Finding Dantzig Selectors

Sorting and Selection with Imprecise Comparisons

Subadditivity and optimal matching of unbounded samples

The Discrete Dantzig Selector: Estimating Sparse Linear Models via Mixed Integer Linear Optimization