Ying Cui,Tim Hoheisel,Tran T. A. Nghia,Defeng Sun
Abstract:In this paper, we study Lipschitz continuity of the solution mappings of regularized least-squares problems for which the convex regularizers have (Fenchel) conjugates that are $\mathcal{C}^2$-cone reducible. Our approach, by using Robinson's strong regularity on the dual problem, allows us to obtain new characterizations of Lipschitz stability that rely solely on first-order information, thus bypassing the need to explore second-order information (curvature) of the regularizer. We show that these solution mappings are automatically Lipschitz continuous around the points in question whenever they are locally single-valued. We leverage our findings to obtain new characterizations of full stability and tilt stability for a broader class of convex additive-composite problems.
What problem does this paper attempt to address?
The problem that this paper attempts to solve is the Lipschitz stability of the regularized least - squares problem, especially when the Fenchel conjugate function of the regularization term is C2 - cone - reducible. Specifically, the author studies whether the solution mapping of these problems is single - valued and Lipschitz continuous under small perturbations.
### Research Background and Motivation
Stability (or sensitivity) analysis in optimization problems studies how the optimal solution (or optimal value) changes when the data (parameters) defining the problem change. This field has a long history, mainly laid by the pioneering work of Kojima [17, 18] and Robinson [28, 29, 30], etc. In recent years, with the wide application of non - smooth convex optimization problems in machine learning and statistical estimation, the stability analysis of such problems has received attention again.
### Core Problems of the Paper
This paper pays special attention to the regularized least - squares problem in the following form:
\[
\min_{x\in X}\frac{1}{2\mu}\|Ax - b\|^{2}+g(x),
\]
where \(A:X\rightarrow Y\) is a linear operator between two Euclidean spaces, \(b\in Y\), \(\mu>0\) is a tuning parameter, and \(g:X\rightarrow\mathbb{R}\cup\{+\infty\}\) is a closed, proper, convex function (usually called a regularization term in machine learning). When regarding the triple \((A, b, \mu)\) as parameters, it is important to study the stability of the solution mapping \(S(A, b, \mu)\) of problem (1) when the initial data \((A, b, \mu)\) has small perturbations.
### Main Contributions
1. **Lipschitz Stability Conditions**: By using Robinson's strong regularity theory, the author gives new characterizations of Lipschitz stability, which only depend on first - order information, thus avoiding the exploration of the curvature (second - order information) of the regularization term.
2. **Single - Valuedness and Lipschitz Continuity**: The author proves that the solution mapping \(S(A, b, \mu)\) is single - valued and Lipschitz continuous under certain conditions.
3. **Extension to Broader Problems**: The author uses these results to obtain new characterizations of complete stability and tilt stability for broader convex additive composite problems.
### Methodology
The author adopts the strong regularity theory and achieves this goal by studying the dual problem. Specifically, they consider the dual problem in the following form:
\[
D(A, b, \mu, v):\min_{y\in Y}\frac{\mu}{2}\|y\|^{2}-\langle b, y\rangle+g^{*}(A^{*}y + v).
\]
By studying the strong regularity of the dual problem, the author can derive the Lipschitz stability conditions of the solution mapping of the original problem. The advantage of this method is that it can simplify the second - order calculation requirements for the regularization term, making the conditions easier to verify.
### Conclusion
The main contribution of this paper is to provide a new method to study the Lipschitz stability of the regularized least - squares problem, especially when the Fenchel conjugate function of the regularization term is C2 - cone - reducible. By introducing the strong regularity theory, the author not only simplifies the expression of the conditions but also expands the application range of these results, covering broader convex optimization problems.