Supplemental: ADMM for High-Dimensional Sparse Penalized Quantile Regression

Yuwen Gu,Jun Fan,Lingchen Kong,Shiqian Ma,Hui Zou

2018-01-01

Technometrics

Abstract:Supplemental material for ADMM for High-Dimensional Sparse Penalized Quantile Regression...

What problem does this paper attempt to address?

Accelerated Stochastic ADMM with Variance Reduction

Chao Zhang,Zebang Shen,Hui Qian,Tengfei Zhou,Jianya Zhou,Jianying Zhou

DOI: https://doi.org/10.48550/arxiv.1611.04074

2016-01-01

Abstract:Alternating Direction Method of Multipliers (ADMM) is a popular method for solving large-scale Machine Learning problems. Stochastic ADMM was proposed to reduce the per iteration computational complexity, which is more suitable for big data problems. Recently, variance reduction techniques have been integrated with stochastic ADMM in order to get a faster convergence rate, such as SAG-ADMM and SVRG-ADMM. However, their convergence rate is still suboptimal w.r.t the smoothness constant. In this paper, we propose an accelerated stochastic ADMM algorithm with variance reduction, which enjoys a faster convergence than all the existing stochastic ADMM algorithms. We theoretically analyse its convergence rate and show its dependence on the smoothness constant is optimal. We also empirically validate its effectiveness and show its priority over other stochastic ADMM algorithms.
Decentralized Smoothing ADMM for Quantile Regression with Non-Convex Sparse Penalties

Reza Mirzaeifard,Diyako Ghaderyan,Stefan Werner

2024-08-09

Abstract:In the rapidly evolving internet-of-things (IoT) ecosystem, effective data analysis techniques are crucial for handling distributed data generated by sensors. Addressing the limitations of existing methods, such as the sub-gradient approach, which fails to distinguish between active and non-active coefficients effectively, this paper introduces the decentralized smoothing alternating direction method of multipliers (DSAD) for penalized quantile regression. Our method leverages non-convex sparse penalties like the minimax concave penalty (MCP) and smoothly clipped absolute deviation (SCAD), improving the identification and retention of significant predictors. DSAD incorporates a total variation norm within a smoothing ADMM framework, achieving consensus among distributed nodes and ensuring uniform model performance across disparate data sources. This approach overcomes traditional convergence challenges associated with non-convex penalties in decentralized settings. We present theoretical proofs and extensive simulation results to validate the effectiveness of the DSAD, demonstrating its superiority in achieving reliable convergence and enhancing estimation accuracy compared with prior methods.

Machine Learning
Locally adaptive sparse additive quantile regression model with TV penalty

Yue Wang,Hongmei Lin,Zengyan Fan,Heng Lian

DOI: https://doi.org/10.1016/j.jspi.2024.106144

IF: 1.095

2024-01-20

Journal of Statistical Planning and Inference

Abstract:High-dimensional additive quantile regression model via penalization provides a powerful tool for analyzing complex data in many contemporary applications. Despite the fast developments, how to combine the strengths of additive quantile regression with total variation penalty with theoretical guarantees still remains unexplored. In this paper, we propose a new methodology for sparse additive quantile regression model over bounded variation function classes via the empirical norm penalty and the total variation penalty for local adaptivity. Theoretically, we prove that the proposed method achieves the optimal convergence rate under mild assumptions. Moreover, an alternating direction method of multipliers (ADMM) based algorithm is developed. Both simulation results and real data analysis confirm the effectiveness of our method.

statistics & probability
High-Dimensional Composite Quantile Regression: Optimal Statistical Guarantees and Fast Algorithms

Haeseong Moon,Wen-Xin Zhou

DOI: https://doi.org/10.1214/23-EJS2147

2023-10-13

Abstract:The composite quantile regression (CQR) was introduced by Zou and Yuan [Ann. Statist. 36 (2008) 1108--1126] as a robust regression method for linear models with heavy-tailed errors while achieving high efficiency. Its penalized counterpart for high-dimensional sparse models was recently studied in Gu and Zou [IEEE Trans. Inf. Theory 66 (2020) 7132--7154], along with a specialized optimization algorithm based on the alternating direct method of multipliers (ADMM). Compared to the various first-order algorithms for penalized least squares, ADMM-based algorithms are not well-adapted to large-scale problems. To overcome this computational hardness, in this paper we employ a convolution-smoothed technique to CQR, complemented with iteratively reweighted $\ell_1$-regularization. The smoothed composite loss function is convex, twice continuously differentiable, and locally strong convex with high probability. We propose a gradient-based algorithm for penalized smoothed CQR via a variant of the majorize-minimization principal, which gains substantial computational efficiency over ADMM. Theoretically, we show that the iteratively reweighted $\ell_1$-penalized smoothed CQR estimator achieves near-minimax optimal convergence rate under heavy-tailed errors without any moment constraint, and further achieves near-oracle convergence rate under a weaker minimum signal strength condition than needed in Gu and Zou (2020). Numerical studies demonstrate that the proposed method exhibits significant computational advantages without compromising statistical performance compared to two state-of-the-art methods that achieve robustness and high efficiency simultaneously.

Methodology
Feature-splitting algorithms for ultrahigh dimensional quantile regression

Jiawei Wen,Songshan Yang,Christina Dan Wang,Yifan Jiang,Runze Li

DOI: https://doi.org/10.1016/j.jeconom.2023.01.028

IF: 3.363

2023-03-26

Journal of Econometrics

Abstract:This paper is concerned with computational issues related to penalized quantile regression (PQR) with ultrahigh dimensional predictors. Various algorithms have been developed for PQR, but they become ineffective and/or infeasible in the presence of ultrahigh dimensional predictors due to the storage and scalability limitations. The variable updating schema of the feature-splitting algorithm that directly applies the ordinary alternating direction method of multiplier (ADMM) to ultrahigh dimensional PQR may make the algorithm fail to converge. To tackle this hurdle, we propose an efficient and parallelizable algorithm for ultrahigh dimensional PQR based on the three-block ADMM. The compatibility of the proposed algorithm with parallel computing alleviates the storage and scalability limitations of a single machine in the large-scale data processing. We establish the rate of convergence of the newly proposed algorithm. In addition, Monte Carlo simulations are conducted to compare the finite sample performance of the proposed algorithm with that of other existing algorithms. The numerical comparison implies that the proposed algorithm significantly outperforms the existing ones. We further illustrate the proposed algorithm via an empirical analysis of a real-world data set.

economics,social sciences, mathematical methods,mathematics, interdisciplinary applications
An Alternating Direction Method of Multipliers for MCP-penalized Regression with High-dimensional Data

Yue Yong Shi,Yu Ling Jiao,Yong Xiu Cao,Yan

DOI: https://doi.org/10.1007/s10114-018-7096-8

2018-01-01

Abstract:The minimax concave penalty (MCP) has been demonstrated theoretically and practically to be effective in nonconvex penalization for variable selection and parameter estimation. In this paper, we develop an efficient alternating direction method of multipliers (ADMM) with continuation algorithm for solving the MCP-penalized least squares problem in high dimensions. Under some mild conditions, we study the convergence properties and the Karush–Kuhn–Tucker (KKT) optimality conditions of the proposed method. A high-dimensional BIC is developed to select the optimal tuning parameters. Simulations and a real data example are presented to illustrate the efficiency and accuracy of the proposed method.
A unified consensus-based parallel ADMM algorithm for high-dimensional regression with combined regularizations

Xiaofei Wu, Zhimin Zhang, Zhenyu Cui

2023-11-21

Abstract:The parallel alternating direction method of multipliers (ADMM) algorithm is widely recognized for its effectiveness in handling large-scale datasets stored in a distributed manner, making it a popular choice for solving statistical learning models. However, there is currently limited research on parallel algorithms specifically designed for high-dimensional regression with combined (composite) regularization terms. These terms, such as elastic-net, sparse group lasso, sparse fused lasso, and their nonconvex variants, have gained significant attention in various fields due to their ability to incorporate prior information and promote sparsity within specific groups or fused variables. The scarcity of parallel algorithms for combined regularizations can be attributed to the inherent nonsmoothness and complexity of these terms, as well as the absence of closed-form solutions for certain proximal operators associated with them. In this paper, we propose a unified constrained optimization formulation based on the consensus problem for these types of convex and nonconvex regression problems and derive the corresponding parallel ADMM algorithms. Furthermore, we prove that the proposed algorithm not only has global convergence but also exhibits linear convergence rate. Extensive simulation experiments, along with a financial example, serve to demonstrate the reliability, stability, and scalability of our algorithm. The R package for implementing the proposed algorithms can be obtained at https://github.com/xfwu1016/CPADMM.

Machine Learning,Statistics Theory
An Empirical Study of ADMM for Nonconvex Problems

Zheng Xu,Soham De,Mario Figueiredo,Christoph Studer,Tom Goldstein

DOI: https://doi.org/10.48550/arXiv.1612.03349

2016-12-10

Optimization and Control

Abstract:The alternating direction method of multipliers (ADMM) is a common optimization tool for solving constrained and non-differentiable problems. We provide an empirical study of the practical performance of ADMM on several nonconvex applications, including l0 regularized linear regression, l0 regularized image denoising, phase retrieval, and eigenvector computation. Our experiments suggest that ADMM performs well on a broad class of non-convex problems. Moreover, recently proposed adaptive ADMM methods, which automatically tune penalty parameters as the method runs, can improve algorithm efficiency and solution quality compared to ADMM with a non-tuned penalty.
Partition-Insensitive Parallel ADMM Algorithm for High-dimensional Linear Models

Xiaofei Wu,Jiancheng Jiang,Zhimin Zhang

2024-04-10

Abstract:The parallel alternating direction method of multipliers (ADMM) algorithms have gained popularity in statistics and machine learning due to their efficient handling of large sample data problems. However, the parallel structure of these algorithms, based on the consensus problem, can lead to an excessive number of auxiliary variables when applied to highdimensional data, resulting in large computational burden. In this paper, we propose a partition-insensitive parallel framework based on the linearized ADMM (LADMM) algorithm and apply it to solve nonconvex penalized high-dimensional regression problems. Compared to existing parallel ADMM algorithms, our algorithm does not rely on the consensus problem, resulting in a significant reduction in the number of variables that need to be updated at each iteration. It is worth noting that the solution of our algorithm remains largely unchanged regardless of how the total sample is divided, which is known as partition-insensitivity. Furthermore, under some mild assumptions, we prove the convergence of the iterative sequence generated by our parallel algorithm. Numerical experiments on synthetic and real datasets demonstrate the feasibility and validity of the proposed algorithm. We provide a publicly available R software package to facilitate the implementation of the proposed algorithm.

Mathematics
Accelerated Nonconvex ADMM with Self-Adaptive Penalty for Rank-Constrained Model Identification

Qingyuan Liu,Zhengchao Huang,Hao Ye,Dexian Huang,Chao Shang

2023-09-08

Abstract:The alternating direction method of multipliers (ADMM) has been widely adopted in low-rank approximation and low-order model identification tasks; however, the performance of nonconvex ADMM is highly reliant on the choice of penalty parameter. To accelerate ADMM for solving rank-constrained identification problems, this paper proposes a new self-adaptive strategy for automatic penalty update. Guided by first-order analysis of the increment of the augmented Lagrangian, the self-adaptive penalty updating enables effective and balanced minimization of both primal and dual residuals and thus ensures a stable convergence. Moreover, improved efficiency can be obtained within the Anderson acceleration scheme. Numerical examples show that the proposed strategy significantly accelerates the convergence of nonconvex ADMM while alleviating the critical reliance on tedious tuning of penalty parameters.

Optimization and Control,Systems and Control
Estimating Functions and Derivatives Via Adaptive Penalized Splines.

Lianqiang Yang,Mengzhen Ding,Yongmiao Hong,Xuejun Wang

DOI: https://doi.org/10.1080/03610918.2019.1594894

2019-01-01

Communications in Statistics - Simulation and Computation

Abstract:Adaptive penalized splines via radial basis are constructed to estimate regression functions and their derivatives. A weight vector based on the range of observations is embedded into the penalty matrix, which remarkably improves the adaptability of the penalized spline smoothing model. Fast computation and comparison with traditional spline models are studied, and the empirical results and simulations show that the new method outperforms smoothing splines, traditional penalized splines and local polynomial smoothing when estimating regression functions and their derivatives, particularly when the observations have inhomogeneous variation.
Penalized weighted smoothed quantile regression for high‐dimensional longitudinal data

Yanan Song,Haohui Han,Liya Fu,Ting Wang

DOI: https://doi.org/10.1002/sim.10056

2024-03-10

Statistics in Medicine

Abstract:Quantile regression, known as a robust alternative to linear regression, has been widely used in statistical modeling and inference. In this paper, we propose a penalized weighted convolution‐type smoothed method for variable selection and robust parameter estimation of the quantile regression with high dimensional longitudinal data. The proposed method utilizes a twice‐differentiable and smoothed loss function instead of the check function in quantile regression without penalty, and can select the important covariates consistently using the efficient gradient‐based iterative algorithms when the dimension of covariates is larger than the sample size. Moreover, the proposed method can circumvent the influence of outliers in the response variable and/or the covariates. To incorporate the correlation within each subject and enhance the accuracy of the parameter estimation, a two‐step weighted estimation method is also established. Furthermore, we prove the oracle properties of the proposed method under some regularity conditions. Finally, the performance of the proposed method is demonstrated by simulation studies and two real examples.

public, environmental & occupational health,medicine, research & experimental,medical informatics,mathematical & computational biology,statistics & probability
Simultaneous Estimation and Dataset Selection for Transfer Learning in High Dimensions by a Non-convex Penalty

Zeyu Li,Dong Liu,Yong He,Xinsheng Zhang

2023-01-01

Abstract:In this paper, we propose to estimate model parameters and identify informative source datasets simultaneously for high-dimensional transfer learning problems with the aid of a non-convex penalty, in contrast to the separate useful dataset selection and transfer learning procedures in the existing literature. To numerically solve the non-convex problem with respect to two specific statistical models, namely the sparse linear regression and the generalized low-rank trace regression models, we adopt the difference of convex (DC) programming with the alternating direction method of multipliers (ADMM) procedures. We theoretically justify the proposed algorithm from both statistical and computational perspectives. Extensive numerical results are reported alongside to validate the theoretical assertions. An package is developed to implement the proposed methods.
ADMM Algorithmic Regularization Paths for Sparse Statistical Machine Learning

Yue Hu,Eric Chi,Genevera I. Allen

DOI: https://doi.org/10.48550/arXiv.1504.06637

2015-04-25

Abstract:Optimization approaches based on operator splitting are becoming popular for solving sparsity regularized statistical machine learning models. While many have proposed fast algorithms to solve these problems for a single regularization parameter, conspicuously less attention has been given to computing regularization paths, or solving the optimization problems over the full range of regularization parameters to obtain a sequence of sparse models. In this chapter, we aim to quickly approximate the sequence of sparse models associated with regularization paths for the purposes of statistical model selection by using the building blocks from a classical operator splitting method, the Alternating Direction Method of Multipliers (ADMM). We begin by proposing an ADMM algorithm that uses warm-starts to quickly compute the regularization path. Then, by employing approximations along this warm-starting ADMM algorithm, we propose a novel concept that we term the ADMM Algorithmic Regularization Path. Our method can quickly outline the sequence of sparse models associated with the regularization path in computational time that is often less than that of using the ADMM algorithm to solve the problem at a single regularization parameter. We demonstrate the applicability and substantial computational savings of our approach through three popular examples, sparse linear regression, reduced-rank multi-task learning, and convex clustering.

Methodology
An ADMM with Continuation Algorithm for Non-Convex SICA-penalized Regression in High Dimensions

Yueyong Shi,Yuanshan Wu,Deyi Xu,Yuling Jiao

DOI: https://doi.org/10.1080/00949655.2018.1448397

2018-01-01

Abstract:The smooth integration of counting and absolute deviation (SICA) penalty has been demonstrated theoretically and practically to be effective in non-convex penalization for variable selection. However, solving the non-convex optimization problem associated with the SICA penalty when the number of variables exceeds the sample size remains to be enriched due to the singularity at the origin and the non-convexity of the SICA penalty function. In this paper, we develop an efficient and accurate alternating direction method of multipliers with continuation algorithm for solving the SICA-penalized least squares problem in high dimensions. We establish the convergence property of the proposed algorithm under some mild regularity conditions and study the corresponding Karush-Kuhn-Tucker optimality condition. A high-dimensional Bayesian information criterion is developed to select the optimal tuning parameters. We conduct extensive simulations studies to evaluate the efficiency and accuracy of the proposed algorithm, while its practical usefulness is further illustrated with a high-dimensional microarray study.
MM for Penalized Estimation

Zhu Wang

DOI: https://doi.org/10.1007/s11749-021-00770-2

2022-03-02

Abstract:Penalized estimation can conduct variable selection and parameter estimation simultaneously. The general framework is to minimize a loss function subject to a penalty designed to generate sparse variable selection. The majorization-minimization (MM) algorithm is a computational scheme for stability and simplicity, and the MM algorithm has been widely applied in penalized estimation. Much of the previous work have focused on convex loss functions such as generalized linear models. When data are contaminated with outliers, robust loss functions can generate more reliable estimates. Recent literature has witnessed a growing impact of nonconvex loss-based methods, which can generate robust estimation for data contaminated with outliers. This article investigates MM algorithm for penalized estimation, provide innovative optimality conditions and establish convergence theory with both convex and nonconvex loss functions. With respect to applications, we focus on several nonconvex loss functions, which were formerly studied in machine learning for regression and classification problems. Performance of the proposed algorithms are evaluated on simulated and real data including healthcare costs and cancer clinical status. Efficient implementations of the algorithms are available in the R package mpath in CRAN.

Computation,Machine Learning
Application of the ADMM Algorithm for a High-Dimensional Partially Linear Model

Aifen Feng,Xiaogai Chang,Youlin Shang,Jingya Fan

DOI: https://doi.org/10.3390/math10244767

IF: 2.4

2022-12-16

Mathematics

Abstract:This paper focuses on a high-dimensional semi-parametric regression model in which a partially linear model is used for the parametric part and the B-spline basis function approach is used to estimate the unknown function for the non-parametric part. Within the framework of this model, the constrained least squares estimation is investigated, and the alternating-direction multiplier method (ADMM) is used to solve the model. The convergence is proved under certain conditions. Finally, numerical simulations are performed and applied to workers' wage data from CPS85. The results show that the ADMM algorithm is very effective in solving high-dimensional partially linear models.

mathematics
AN EXTENDED LINEARIZED ALTERNATING DIRECTION METHOD OF MULTIPLIERS FOR FUSED-LASSO PENALIZED LINEAR REGRESSION

Jia Liu,Youlin Shang,Zhengfen Jin,R. O. X. I. N. Zhang

DOI: https://doi.org/10.3934/jimo.2023030

2023-01-01

Journal of Industrial and Management Optimization

Abstract:In terms of variable selection and dimensionality reduction for high-dimensional data, Fused-LASSO is a powerful model when the number of important variables is small and neighboring variables exhibit a high degree of correlation. However, solving the model is challenging due to its non smoothness and non-separability. Alternating direction method of multipliers (ADMM) is an effective method to solve the Fused-LASSO, but one of the subproblems may have no closed-form solution leading to additional computational overhead, and the computing for the inverse of matrix in the subproblem is expensive. In order to overcome these shortcomings and have a great speed performance, an extended linearized alternating direction method of multipliers (ELADMM) is proposed in this paper. First, in the framework of ADMM, we embed the linearized technique into the x-subproblem, so that the subproblem can be solved quickly and efficiently rather than by inner iteration. Second, we make additional extensions to some variables after each iteration, which improves the speed performance of the LADMM. Numerical results show that the proposed ELADMM outperforms ADMM as well as other existing solvers in the sense that it finds the same accurate variable selection results with the shortest time.
Nonparametric screening for additive quantile regression in ultra-high dimension

Daoji Li,Yinfei Kong,Dawit Zerom

DOI: https://doi.org/10.1080/10485252.2024.2366978

2024-06-20

Journal of Nonparametric Statistics

Abstract:In practical applications, one often does not know the 'true' structure of the underlying conditional quantile function, especially in the ultra-high dimensional setting. To deal with ultra-high dimensionality, quantile-adaptive marginal nonparametric screening methods have been recently developed. However, these approaches may miss important covariates that are marginally independent of the response, or may select unimportant covariates due to their high correlations with important covariates. To mitigate such shortcomings, we develop a conditional nonparametric quantile screening procedure (complemented by subsequent selection) for nonparametric additive quantile regression models. Under some mild conditions, we show that the proposed screening method can identify all relevant covariates in a small number of steps with probability approaching one. The subsequent narrowed best subset (via a modified Bayesian information criterion) also contains all the relevant covariates with overwhelming probability. The advantages of our proposed procedure are demonstrated through simulation studies and a real data example.

statistics & probability
Accelerated stochastic admm for empirical risk minimization

Chao Zhang,Qian Hui,Zebang Shen,tengfei Zhou

2016-01-01

Abstract:Alternating Direction Method of Multipliers (ADMM) is a popular method in solving Machine Learning problems. Stochastic ADMM was firstly proposed in order to reduce the per iteration computational complexity, which is more suitable for big data problems. Recently, variance reduction techniques have been integrated with stochastic ADMM in order to get a fast convergence rate, such as SAG-ADMM and SVRG-ADMM, but the convergence is still suboptimal wrt the smoothness constant. In this paper, we propose a new accelerated stochastic ADMM algorithm with variance reduction, which enjoys a faster convergence than all the other stochastic ADMM algorithms. We theoretically analyze its convergence rate and show its dependence on the smoothness constant is optimal. We also empirically validate its effectiveness and show its priority over other stochastic ADMM algorithms.

Supplemental: ADMM for High-Dimensional Sparse Penalized Quantile Regression

Accelerated Stochastic ADMM with Variance Reduction

Decentralized Smoothing ADMM for Quantile Regression with Non-Convex Sparse Penalties

Locally adaptive sparse additive quantile regression model with TV penalty

High-Dimensional Composite Quantile Regression: Optimal Statistical Guarantees and Fast Algorithms

Feature-splitting algorithms for ultrahigh dimensional quantile regression

An Alternating Direction Method of Multipliers for MCP-penalized Regression with High-dimensional Data

A unified consensus-based parallel ADMM algorithm for high-dimensional regression with combined regularizations

An Empirical Study of ADMM for Nonconvex Problems

Partition-Insensitive Parallel ADMM Algorithm for High-dimensional Linear Models

Accelerated Nonconvex ADMM with Self-Adaptive Penalty for Rank-Constrained Model Identification

Estimating Functions and Derivatives Via Adaptive Penalized Splines.

Penalized weighted smoothed quantile regression for high‐dimensional longitudinal data

Simultaneous Estimation and Dataset Selection for Transfer Learning in High Dimensions by a Non-convex Penalty

ADMM Algorithmic Regularization Paths for Sparse Statistical Machine Learning

An ADMM with Continuation Algorithm for Non-Convex SICA-penalized Regression in High Dimensions

MM for Penalized Estimation

Application of the ADMM Algorithm for a High-Dimensional Partially Linear Model

AN EXTENDED LINEARIZED ALTERNATING DIRECTION METHOD OF MULTIPLIERS FOR FUSED-LASSO PENALIZED LINEAR REGRESSION

Nonparametric screening for additive quantile regression in ultra-high dimension

Accelerated stochastic admm for empirical risk minimization