Abstract:In this paper, we study a variant of the quadratic penalty method for linearly constrained convex problems, which has already been widely used but actually lacks theoretical justification. Namely, the penalty parameter steadily increases and the penalized objective function is minimized inexactly rather than exactly, e.g., with only one step of the proximal gradient descent. For such a variant of the quadratic penalty method, we give counterexamples to show that it may not give a solution to the original constrained problem. By choosing special penalty parameters, we ensure the convergence and further establish the convergence rates of $O\left(\frac{1}{\sqrt{K}}\right)$ for the generally convex problems and $O\left(\frac{1}{K}\right)$ for strongly convex ones, where $K$ is the number of iterations. Furthermore, by adopting Nesterov's extrapolation we show that the convergence rates can be improved to $O\left(\frac{1}{K}\right)$ for the generally convex problems and $O\left(\frac{1}{K^2}\right)$ for strongly convex ones. When applied to the decentralized distributed optimization, the penalty methods studied in this paper become the widely used distributed gradient method and the fast distributed gradient method. However, due to the totally different analysis framework, we can improve their $O\left(\frac{\log K}{\sqrt{K}}\right)$ and $O\left(\frac{\log K}{K}\right)$ convergence rates to $O\left(\frac{1}{\sqrt{K}}\right)$ and $O\left(\frac{1}{K}\right)$ with fewer assumptions on the network topology for general convex problems. Using our analysis framework, we also extend the fast distributed gradient method to a communication efficient version, i.e., finding an $\varepsilon$ solution in $O\left(\frac{1}{\varepsilon}\right)$ communications and $O\left(\frac{1}{\varepsilon^{2+\delta}}\right)$ computations for the non-smooth problems, where $\delta$ is a small constant.

Convergence Rate for Nonparametric Quantile Regression with a Total Variation Penalty

Nonparametric Quantile Regression: Non-Crossing Constraints and Conformal Prediction

Locally adaptive sparse additive quantile regression model with TV penalty

Sparse quantile regression

Nearly Semiparametric Efficient Estimation of Quantile Regression

Non-crossing convex quantile regression

Convergence in total variation for the kinetic Langevin algorithm

Quantile regression for compositional covariates

Sparse Estimation Via Lower-Order Penalty Optimization Methods in High-Dimensional Linear Regression.

Improved convergence rates of nonparametric penalized regression under misspecified total variation

Statistical Performance of Quantile Tensor Regression with Convex Regularization

An Empirical Quantile Estimation Approach for Chance-Constrained Nonlinear Optimization Problems

An Empirical Quantile Estimation Approach to Nonlinear Optimization Problems with Chance Constraints

Local convergence rates of the nonparametric least squares estimator with applications to transfer learning

Online Quantile Regression

Convergence analysis of quasi-Monte Carlo sampling for quantile and expected shortfall

Minimax rates of convergence for nonparametric location-scale models

Quantitative Convergence of Quadratically Regularized Linear Programs

Quantile Regression in Partially Linear Varying Coefficient Models

Convergence Rates Analysis of The Quadratic Penalty Method and Its Applications to Decentralized Distributed Optimization

Partial functional linear quantile regression