Lipschitz stability of least-squares problems regularized by functions with $\mathcal{C}^2$-cone reducible conjugates

Ying Cui,Tim Hoheisel,Tran T. A. Nghia,Defeng Sun
2024-09-20
Abstract:In this paper, we study Lipschitz continuity of the solution mappings of regularized least-squares problems for which the convex regularizers have (Fenchel) conjugates that are $\mathcal{C}^2$-cone reducible. Our approach, by using Robinson's strong regularity on the dual problem, allows us to obtain new characterizations of Lipschitz stability that rely solely on first-order information, thus bypassing the need to explore second-order information (curvature) of the regularizer. We show that these solution mappings are automatically Lipschitz continuous around the points in question whenever they are locally single-valued. We leverage our findings to obtain new characterizations of full stability and tilt stability for a broader class of convex additive-composite problems.
Optimization and Control
What problem does this paper attempt to address?
The problem that this paper attempts to solve is the Lipschitz stability of the regularized least - squares problem, especially when the Fenchel conjugate function of the regularization term is C2 - cone - reducible. Specifically, the author studies whether the solution mapping of these problems is single - valued and Lipschitz continuous under small perturbations. ### Research Background and Motivation Stability (or sensitivity) analysis in optimization problems studies how the optimal solution (or optimal value) changes when the data (parameters) defining the problem change. This field has a long history, mainly laid by the pioneering work of Kojima [17, 18] and Robinson [28, 29, 30], etc. In recent years, with the wide application of non - smooth convex optimization problems in machine learning and statistical estimation, the stability analysis of such problems has received attention again. ### Core Problems of the Paper This paper pays special attention to the regularized least - squares problem in the following form: \[ \min_{x\in X}\frac{1}{2\mu}\|Ax - b\|^{2}+g(x), \] where \(A:X\rightarrow Y\) is a linear operator between two Euclidean spaces, \(b\in Y\), \(\mu>0\) is a tuning parameter, and \(g:X\rightarrow\mathbb{R}\cup\{+\infty\}\) is a closed, proper, convex function (usually called a regularization term in machine learning). When regarding the triple \((A, b, \mu)\) as parameters, it is important to study the stability of the solution mapping \(S(A, b, \mu)\) of problem (1) when the initial data \((A, b, \mu)\) has small perturbations. ### Main Contributions 1. **Lipschitz Stability Conditions**: By using Robinson's strong regularity theory, the author gives new characterizations of Lipschitz stability, which only depend on first - order information, thus avoiding the exploration of the curvature (second - order information) of the regularization term. 2. **Single - Valuedness and Lipschitz Continuity**: The author proves that the solution mapping \(S(A, b, \mu)\) is single - valued and Lipschitz continuous under certain conditions. 3. **Extension to Broader Problems**: The author uses these results to obtain new characterizations of complete stability and tilt stability for broader convex additive composite problems. ### Methodology The author adopts the strong regularity theory and achieves this goal by studying the dual problem. Specifically, they consider the dual problem in the following form: \[ D(A, b, \mu, v):\min_{y\in Y}\frac{\mu}{2}\|y\|^{2}-\langle b, y\rangle+g^{*}(A^{*}y + v). \] By studying the strong regularity of the dual problem, the author can derive the Lipschitz stability conditions of the solution mapping of the original problem. The advantage of this method is that it can simplify the second - order calculation requirements for the regularization term, making the conditions easier to verify. ### Conclusion The main contribution of this paper is to provide a new method to study the Lipschitz stability of the regularized least - squares problem, especially when the Fenchel conjugate function of the regularization term is C2 - cone - reducible. By introducing the strong regularity theory, the author not only simplifies the expression of the conditions but also expands the application range of these results, covering broader convex optimization problems.