Abstract:This note extends the results of classical parametric statistics like Fisher and Wilks theorem to modern setups with a high or infinite parameter dimension, limited sample size, and possible model misspecification. We consider a special class of stochastically linear smooth (SLS) models satisfying three major conditions: the stochastic component of the log-likelihood is linear in the model parameter and the expected log-likelihood is a smooth and concave function. For the penalized maximum likelihood estimators (pMLE), we establish three types of results: (1) concentration in a small vicinity of the ``truth''; (2) Fisher and Wilks expansions; (3) risk bounds. In all results, the remainder is given explicitly and can be evaluated in terms of the effective sample size and effective parameter dimension which allows us to identify the so-called \emph{critical parameter dimension}. The results are also dimension and coordinate-free. The obtained finite sample expansions are of special interest because they can be used not only for obtaining the risk bounds but also for inference, studying the asymptotic distribution, analysis of resampling procedures, etc. The main tool for all these expansions is the so-called ``basic lemma'' about linearly perturbed optimization. Despite their generality, all the presented bounds are nearly sharp and the classical asymptotic results can be obtained as simple corollaries. Our results indicate that the use of advanced fourth-order expansions allows to relax the critical dimension condition $ \mathbb{p}^{3} \ll n $ from Spokoiny (2023a) to $ \mathbb{p}^{3/2} \ll n $. Examples for classical models like logistic regression, log-density and precision matrix estimation illustrate the applicability of general results.

Scale-sensitive Psi-dimensions: the Capacity Measures for Classifiers Taking Values in R^Q

Optimal Multivariate Quota-Share Reinsurance: A Nonparametric Mean-Cvar Framework

Probabilistic Safety Regions Via Finite Families of Scalable Classifiers

Worst-Case Range Value-at-Risk with Partial Information

Error Bounds for Real Function Classes Based on Discretized Vapnik-Chervonenkis Dimensions.

On the Capacity Limits of Privileged ERM

Risk quantization by magnitude and propensity

Minimax risk classifiers with 0-1 loss

Dimensionality Dependent PAC-Bayes Margin Bound.

Generalization Bounds for Vicinal Risk Minimization Principle

High-dimensional classification by sparse logistic regression

Sensitivity estimation of conditional value at risk using randomized quasi-Monte Carlo

Error Scaling Laws for Kernel Classification under Source and Capacity Conditions

Risk-Averse Classification

Finite sample expansions and risk bounds in high-dimensional SLS models

Maxitive monetary risk measures: worst-case risk assessment and sharp large deviations

Distribution-free Deviation Bounds and The Role of Domain Knowledge in Learning via Model Selection with Cross-validation Risk Estimation

Multivariate risks and depth-trimmed regions

Disentangling price, risk and model risk: V&R measures

Information-Theoretic Bayes Risk Lower Bounds for Realizable Models

Uncertainty quantification in metric spaces