Abstract:The composite quantile regression (CQR) was introduced by Zou and Yuan [Ann. Statist. 36 (2008) 1108--1126] as a robust regression method for linear models with heavy-tailed errors while achieving high efficiency. Its penalized counterpart for high-dimensional sparse models was recently studied in Gu and Zou [IEEE Trans. Inf. Theory 66 (2020) 7132--7154], along with a specialized optimization algorithm based on the alternating direct method of multipliers (ADMM). Compared to the various first-order algorithms for penalized least squares, ADMM-based algorithms are not well-adapted to large-scale problems. To overcome this computational hardness, in this paper we employ a convolution-smoothed technique to CQR, complemented with iteratively reweighted $\ell_1$-regularization. The smoothed composite loss function is convex, twice continuously differentiable, and locally strong convex with high probability. We propose a gradient-based algorithm for penalized smoothed CQR via a variant of the majorize-minimization principal, which gains substantial computational efficiency over ADMM. Theoretically, we show that the iteratively reweighted $\ell_1$-penalized smoothed CQR estimator achieves near-minimax optimal convergence rate under heavy-tailed errors without any moment constraint, and further achieves near-oracle convergence rate under a weaker minimum signal strength condition than needed in Gu and Zou (2020). Numerical studies demonstrate that the proposed method exhibits significant computational advantages without compromising statistical performance compared to two state-of-the-art methods that achieve robustness and high efficiency simultaneously.
What problem does this paper attempt to address?
The problems that this paper attempts to solve are the computational efficiency and statistical performance issues encountered when performing Composite Quantile Regression (CQR) in high - dimensional data. Specifically:
1. **Computational efficiency problem**: Traditional CQR methods have a high computational cost when dealing with large - scale data sets, especially when using optimization algorithms such as the Alternating Direction Method of Multipliers (ADMM). Although the ADMM algorithm is effective, it performs poorly on large - scale problems, and the computational complexity of each step of its update is \(O(pnq+(p + q)^2)\), where \(p\) is the number of variables, \(n\) is the sample size, and \(q\) is the number of quantile levels used. This high computational complexity limits the application of CQR to large - scale data.
2. **Statistical performance problem**: Under heavy - tailed error distributions, traditional least - squares regression methods (such as LASSO) are very sensitive to the tails of the error distribution, which may lead to inaccurate estimations. Although the CQR method performs well when dealing with heavy - tailed errors and can still maintain good asymptotic efficiency under error distribution moment - free conditions (i.e., allowing for infinite variance), its computational complexity is high, especially in high - dimensional sparse models. In addition, the adaptability of the CQR method under strong signals is not as good as some non - convex penalty methods (such as SCAD or MC penalty).
To overcome these problems, the paper proposes the Smoothed Composite Quantile Regression (SCQR) method with convolution smoothing, combined with iterative re - weighted \(L_1\) regularization to fit sparse models. By introducing convolution smoothing techniques, the SCQR method not only improves computational efficiency but also maintains statistical performance. Specific improvements include:
- **Convolution smoothing**: Convert the piecewise - linear check function into a twice continuously differentiable, convex, and locally strongly convex surrogate function, thereby simplifying the optimization problem.
- **Iterative re - weighted \(L_1\) regularization**: Through iterative re - weighting of \(L_1\) regularization, reduce the bias and improve the adaptability to strong signals.
- **Locally Adaptive Majorization Minimization (LAMM) algorithm**: Compared with ADMM, LAMM is a simpler gradient - based algorithm, which is especially suitable for large - scale problems, and the main computational cost per step is only matrix - vector multiplication.
Through these improvements, the SCQR method significantly improves computational efficiency while maintaining high statistical performance, making it suitable for large - scale high - dimensional data sets.