Abstract:Major progress has been made in the previous decade to characterize the asymptotic behavior of regularized M-estimators in high-dimensional regression problems in the proportional asymptotic regime where the sample size $n$ and the number of features $p$ are increasing simultaneously such that $n/p\to \delta \in(0,\infty)$, using powerful tools such as Approximate Message Passing or the Convex Gaussian Min-Max Theorem (CGMT). The asymptotic error and behavior of the regularized M-estimator is then typically described by a system of nonlinear equations with a few scalar unknowns, and the solution to this system precisely characterizes the asymptotic error. Application of the CGMT and related machinery requires the existence and uniqueness of a solution to this low-dimensional system of equations or to a related scalar convex minimization problem. This paper resolves the question of existence and uniqueness of solution to this low-dimensional system for the case of linear models with independent additive noise, when both the data-fitting loss function and regularizer are separable and convex. Such existence result was previously known under strong convexity or for specific estimators such as the Lasso. The main idea behind this existence result is inspired by an argument developed by Montanari et al. [2023] and Celentano et al. [2023] in different contexts: By constructing an ad-hoc convex minimization problem in an infinite dimensional Hilbert space, the existence of the Lagrange multiplier for this optimization problem makes it possible to construct explicit solutions to the low-dimensional system of interest. The conditions under which we derive this existence result exactly correspond to the side of the phase transition where perfect recovery $\hat{x} = x_0$ fails, so that these conditions are optimal.

On Solvability of an Equation Arising in the Theory of M-Estimates

THE SOLVABILITY FOR NONLINEAR SINGULARLY PERTURBED DIFFERENTIAL SYSTEM

On the Limiting Behaviors of Increments of Sums of Random-Variables Without Moment Conditions

On an Estimate of the Decay Rate for Applications of Razumikhin-type Theorems

Error Bounds for Approximate Solutions of Abstract Inequality Systems and Infinite Systems of Inequalities on Banach Spaces

Limiting Behavior of RecursiveM-Estimators in Multivariate Linear Regression Models

Solvability of Equilibrium Riccati Equations: A Direct Approach

On solvability of an indefinite Riccati equation

Limiting behavior of recursive M-estimators in multivariate linear regression models and their asymptotic efficiencies

Existence of solutions to the nonlinear equations characterizing the precise error of M-estimators

On Increments of Sums of Random-Variables Without Moment Hypotheses

Solvability of a class of systems of quadratic integral equations

The Minimization of Piecewise Functions: Pseudo Stationarity

On Estimation and Optimization of Mean Values of Bounded Variables

Estimates on moments of the solutions to stochastic differential equations with respect to martingales in the plane

Solvability of Second-Order M-Point Difference Equation Boundary Value Problems on Infinite Intervals

Inference on estimators defined by mathematical programming

On the Symmetry of Limiting Distributions of M-estimators

Multi-Hölder Estimates of Solutions to Elliptic Equations of Higher Order

Solvability of a Class of Singular Fourth Order Equations of Monge–Ampère Type

Functional and Operatorial Equations Defined Implicitly and Moment Problems