An extension of McDiarmid's inequality

Richard Combes

2024-05-02

Abstract:We generalize McDiarmid's inequality for functions with bounded differences on a high probability set, using an extension argument. Those functions concentrate around their conditional expectations. We further extend the results to concentration in general metric spaces.

Machine Learning,Probability,Statistics Theory

What problem does this paper attempt to address?

On Bernstein Type Inequalities for Stochastic Integrals of Multivariate Point Processes

Hanchao Wang,Zhengyan Lin,Zhonggen Su

DOI: https://doi.org/10.1016/j.spa.2018.05.014

IF: 1.43

2019-01-01

Stochastic Processes and their Applications

Abstract:In this paper, we first obtain a Bernstein type of concentration inequality for stochastic integrals of multivariate point processes under some conditions through the Doléans-Dade exponential formula, and then derive a uniform exponential inequality using a generic chaining argument. As a direct consequence, we obtain an upper bound for a sequence of discrete time martingales indexed by a class of functionals. Finally, we apply the uniform exponential bound to nonparametric maximum likelihood estimators and provide a rate of convergence in terms of Hellinger distance, which is an improvement of earlier work of van de Geer (1995).
Distribution-dependent McDiarmid-type Inequalities for Functions of Unbounded Interaction.

Shaojie Li,Yong Liu

2023-01-01

Abstract:The concentration of measure inequalities serves an essential role in statistics and machine learning. This paper gives unbounded analogues of the McDiarmid-type exponential inequalities for three popular classes of distributions, namely sub-Gaussian, sub-exponential and heavy-tailed distributions. The inequalities in the sub-Gaussian and sub-exponential cases are distribution-dependent compared with the recent results, and the inequalities in the heavy-tailed case are not available in the previous works. The usefulness of the inequalities is illustrated through applications to the sample mean, U-statistics and V-statistics.
Distribution-dependent Concentration Inequalities for Tighter Generalization Bounds

Xinxing Wu,Junping Zhang

DOI: https://doi.org/10.1007/s11432-017-9225-2

2018-01-01

Science China Information Sciences

Abstract:Concentration inequalities are indispensable tools for studying the generalization capacity of learning models. Hoeffding's and McDiarmid's inequalities are commonly used, giving bounds independent of the data distribution. Although this makes them widely applicable, a drawback is that the bounds can be too loose in some specific cases. Although efforts have been devoted to improving the bounds, we find that the bounds can be further tightened in some distribution-dependent scenarios and conditions for the inequalities can be relaxed. In particular, we propose four types of conditions for probabilistic boundedness and bounded differences, and derive several distribution-dependent extensions of Hoeffding's and McDiarmid's inequalities. These extensions provide bounds for functions not satisfying the conditions of the existing inequalities, and in some special cases, tighter bounds. Furthermore, we obtain generalization bounds for unbounded and hierarchy-bounded loss functions. Finally we discuss the potential applications of our extensions to learning theory.
Exponential inequalities in probability spaces revisited

Ali Barki,Sergey G.Bobkov,Esther Bou Dagher,Cyril Roberto

2023-12-18

Abstract:We revisit several results on exponential integrability in probability spaces and derive some new ones. In particular, we give a quantitative form of recent results by Cianchi-Musil and Pick in the framework of Moser-Trudinger-type inequalities, and recover Ivanisvili-Russell's inequality for the Gaussian measure. One key ingredient is the use of a dual argument, which is new in this context, that we also implement in the discrete setting of the Poisson measure on integers.

Probability,Functional Analysis
Concentration Inequalities for General Functions of Heavy-Tailed Random Variables

Shaojie Li,Yong Liu

2024-01-01

Abstract:Concentration inequalities play an essential role in the study of machine learning and high dimensional statistics. In this paper, we obtain unbounded analogues of the popular bounded difference inequality for functions of independent random variables with heavy-tailed distributions. The main results provide a general framework applicable to all heavy-tailed distributions with finite variance. To illustrate the strength of our results, we present applications to sub-exponential tails, sub-Weibull tails, and heavier polynomially decaying tails. Applied to some standard problems in statistical learning theory (vector valued concentration, Rademacher complexity, and algorithmic stability), we show that these inequalities allow an extension of existing results to heavy-tailed distributions up to finite variance.
Asymmetric Doob Inequalities in Continuous Time

Guixiang Hong,Marius Junge,Javier Parcet

DOI: https://doi.org/10.1016/j.jfa.2017.05.001

IF: 1.891

2017-01-01

Journal of Functional Analysis

Abstract:The present paper is devoted to the second part of our project on asymmetric maximal inequalities, where we consider martingales in continuous time. Let (M, tau) be a noncommutative probability space equipped with a continuous filtration of von Neumann subalgebras (M-t)o <= t <= 1 whose union is weak-* dense in M. Let epsilon(t) denote the corresponding family of conditional expectations. As for discrete filtrations, we shall prove that for 1 < p < 2 and x is an element of L-p(M,tau) one can find a, b is an element of L-p (M, tau) and contractions at, upsilon(t) is an element of M such thatepsilon(t)(x) = au(t) + upsilon(t)b and max {parallel to a parallel to(p),parallel to b parallel to(p)} <= c(p) parallel to x parallel to(p)Moreover, aut and upsilon(t)b converge in the row/column Hardy spaces H-p(r)(M) and H-p(c)(M) respectively. We also confirm in the continuous setting the validity of related asymmetric maximal inequalities which we recently found for discrete filtrations, including p = 1. As for other results in noncommutative martingale theory, the passage from discrete to continuous index is quite technical and requires genuinely new methods. Our approach towards asymmetric maximal inequalities is based on certain construction of conditional expectations for a sequence of projective systems of L-p-modules. The convergence in H-p(r)(M) and H-p(c)(M) also imposes new algebraic atomic decompositions. (C) 2017 Elsevier Inc. All rights reserved.
On a version of a multivariate integration by parts formula for Lebesgue integrals

Jonathan Ansari

2024-08-17

Abstract:Multidimensional integration by parts formulas apply under the standard assumption that one of the functions is continuous and the other has bounded Hardy-Krause variation. Motivated by recently developed results in the probabilistic context of price and risk bounds, this paper provides a version of an integration by parts formula for the Lebesgue integral of measure-inducing functions which may both be discontinuous and may have infinite Hardy-Krause variation. To this end, we give a general definition of measure-inducing functions and establish various of their properties, such as a characterization in terms of Delta-monotone functions. As a consequence of the integration by parts formula, several convergence results are provided, allowing an extension of the Lebesgue integral of a measure-inducing function to the case where one integrates with respect to a continuous semi-copula. The latter class of aggregation functions includes quasi-copulas which serve as bounds for the dependence structure in many applications.

Probability,Classical Analysis and ODEs
A uniform Dvoretzky-Kiefer-Wolfowitz inequality

Daniel Bartl,Shahar Mendelson

2023-12-11

Abstract:We show that under minimal assumption on a class of functions $\mathcal{H}$ defined on a probability space $(\mathcal{X},\mu)$, there is a threshold $\Delta_0$ satisfying the following: for every $\Delta\geq\Delta_0$, with probability at least $1-2\exp(-c\Delta m)$ with respect to $\mu^{\otimes m}$, \[ \sup_{h\in\mathcal{H}} \sup_{t\in\mathbb{R}} \left| \mathbb{P}(h(X)\leq t) - \frac{1}{m}\sum_{i=1}^m 1_{(-\infty,t]}(h(X_i)) \right| \leq \sqrt{\Delta};\] here $X$ is distributed according to $\mu$ and $(X_i)_{i=1}^m$ are independent copies of $X$. The value of $\Delta_0$ is determined by an unexpected complexity parameter of the class $\mathcal{H}$ that captures the set's geometry (Talagrand's $\gamma_1$-functional). The bound, the probability estimate and the value of $\Delta_0$ are all optimal up to a logarithmic factor.

Probability
Bernstein-type and Bennett-type inequalities for unbounded matrix martingales

Alexey Kroshnin,Alexandra Suvorikova

2024-11-12

Abstract:We derive explicit Bernstein-type and Bennett-type concentration inequalities for matrix-valued supermartingale processes with unbounded observations. Specifically, we assume that the $\psi_{\alpha}$-Orlicz (quasi-)norms of their difference process are bounded for some $\alpha > 0$. As corollaries, we prove an empirical version of Bernstein's inequality and an extension of the bounded differences inequality, also known as McDiarmid's inequality.

Probability,Statistics Theory
Concentration inequalities for locally small increments of compound empirical processes with applications to solutions of compound and risk averse stochastical programming

Volker Kratschmer

2024-08-12

Abstract:The paper deals with concentration inequalities for locally small increments of compound empirical processes. In the asymptotic theory of $m$-estimation such inequalities play an essential role in deriving convergence rates for solutions of the sample average approximation method to solve compound stochastic programs, and in particular for $m$-estimators. We develop inequalities dependent on the sample sizes with explicit terms instead of unspecified universal constants. They are applied to study the Sample Average Approximation method for compound stochastic programs. Nonasymptotic upper estimates for the deviation probabilities of the optimal solutions are derived which are dependent on the sample sizes. They allow to conclude immediately convergence rates for the optimal solutions. In the special case of classical risk neutral stochastic programs, we end up with upper estimates of deviation probabilities for $m$-estimators, and their convergence rates. Moreover, we may also demonstrate how to apply the results to sample average approximation of risk averse stochastic programs. In this respect we consider stochastic programs expressed in terms of absolute semideviations and Average Value at Risk. The investigations are based on concentration inequalities from the recent contribution Kratschmer(2024).

Optimization and Control,Probability
Hoeffding's inequality for supermartingales

Xiequan Fan,Ion Grama,Quansheng Liu

DOI: https://doi.org/10.48550/arXiv.1109.4359

2012-07-28

Abstract:We give an extension of Hoeffding's inequality to the case of supermartingales with differences bounded from above. Our inequality strengthens or extends the inequalities of Freedman, Bernstein, Prohorov, Bennett and Nagaev.

Probability
Concentration Inequalities for Sums of Markov Dependent Random Matrices

Joe Neeman,Bobby Shi,Rachel Ward

2023-10-17

Abstract:We give Hoeffding and Bernstein-type concentration inequalities for the largest eigenvalue of sums of random matrices arising from a Markov chain. We consider time-dependent matrix-valued functions on a general state space, generalizing previous that had only considered Hoeffding-type inequalities, and only for time-independent functions on a finite state space. In particular, we study a kind of noncommutative moment generating function, give tight bounds on it, and use a method of Garg et al. to turn this into tail bounds. Our proof proceeds spectrally, bounding the norm of a certain perturbed operator. In the process we make an interesting connection to dynamical systems and Banach space theory to prove a crucial result on the limiting behavior of our moment generating function that may be of independent interest.

Probability
A Doob-type maximal inequality and its applications to various stochastic processes

János Engländer

DOI: https://doi.org/10.48550/arXiv.1805.07700

2018-05-20

Probability

Abstract:We generalize the notion of the submartingale property and Doob's inequality. Furthermore, we show how the latter leads to new inequalities for several stochastic processes: certain time series, Levy processes, random walks, processes with independent increments, branching processes and continuous state branching processes, branching diffusions and superdiffusions, as well as some Markov processes, including geometric Brownian motion.
A short proof of the Dvoretzky--Kiefer--Wolfowitz--Massart inequality

Henry W J Reeve

2024-03-25

Abstract:The Dvoretzky--Kiefer--Wolfowitz--Massart inequality gives a sub-Gaussian tail bound on the supremum norm distance between the empirical distribution function of a random sample and its population counterpart. We provide a short proof of a result that improves the existing bound in two respects. First, our one-sided bound holds without any restrictions on the failure probability, thereby verifying a conjecture of Birnbaum and McCarty (1958). Second, it is local in the sense that it holds uniformly over sub-intervals of the real line with an error rate that adapts to the behaviour of the population distribution function on the interval.

Probability,Statistics Theory
A Matrix Bernstein Inequality for Strong Rayleigh Distributions

Tarun Kathuria

DOI: https://doi.org/10.48550/arXiv.2011.13340

2020-11-26

Abstract:The Entropy method provides a powerful framework for proving scalar concentration inequalities by establishing functional inequalities like Poincare and log-Sobolev inequalities. These inequalities are especially useful for deriving concentration for dependent distributions coming from stationary distributions of Markov chains. In contrast to the scalar case, the study of matrix concentration inequalities is fairly recent and there does not seem to be a general framework for proving matrix concentration inequalities using the Entropy method. In this paper, we initiate the study of Matrix valued functional inequalities and show how matrix valued functional inequalities can be used to establish a kind of iterative variance comparison argument to prove Matrix Bernstein like inequalities. Strong Rayleigh distributions are a class of distributions which satisfy a strong form of negative dependence properties. We show that scalar Poincare inequalities recently established for a flip-swap markov chain considered in the work of Hermon and Salez [HS19] can be lifted to prove matrix valued Poincare inequalities. This can then be combined with the iterative variance argument to prove a version of matrix Bernstein inequality for Lipshitz matrix-valued functions for Strong Rayleigh measures.

Probability
Stability for the Brunn-Minkowski and Riesz Rearrangement Inequalities, with Applications to Gaussian Concentration and Finite Range Non-local Isoperimetry

Eric Carlen,Francesco Maggi

DOI: https://doi.org/10.4153/cjm-2016-026-9

2017-10-01

Canadian Journal of Mathematics

Abstract:Abstract We provide a simple, general argument to obtain improvements of concentration-type inequalities starting from improvements of their corresponding isoperimetric-type inequalities. We apply this argument to obtain robust improvements of the Brunn-Minkowski inequality (for Minkowski sums between generic sets and convex sets) and of the Gaussian concentration inequality. The former inequality is then used to obtain a robust improvement of the Riesz rearrangement inequality under certain natural conditions. These conditions are compatible with the applications to a finite-range nonlocal isoperimetric problem arising in statistical mechanics
The extended Ville's inequality for nonintegrable nonnegative supermartingales

Hongjian Wang,Aaditya Ramdas

2024-10-09

Abstract:Following the initial work by Robbins, we rigorously present an extended theory of nonnegative supermartingales, requiring neither integrability nor finiteness. In particular, we derive a key maximal inequality foreshadowed by Robbins, which we call the extended Ville's inequality, that strengthens the classical Ville's inequality (for integrable nonnegative supermartingales), and also applies to our nonintegrable setting. We derive an extension of the method of mixtures, which applies to $\sigma$-finite mixtures of our extended nonnegative supermartingales. We present some implications of our theory for sequential statistics, such as the use of improper mixtures (priors) in deriving nonparametric confidence sequences and (extended) e-processes.

Probability,Statistics Theory,Methodology,Machine Learning
A generalization of van der Corput's Difference Theorem

Sohail Farhangi

2024-04-24

Abstract:We prove a generalization of van der Corput's Difference Theorem in the theory of uniform distribution by establishing a connection with unitary operators that have Lebesgue spectrum. This allows us to show, for example, that if $(x_n)_{n = 1}^\infty \subseteq [0,1]$ is such that $(x_{n+h}-x_n)_{n = 1}^\infty$ is uniformly distributed for all $h \in \mathbb{N}$, then $(x_{n_k})_{k = 1}^\infty$ is uniformly distributed, where $(n_k)_{k = 1}^\infty$ is an enumeration of the $1s$ in the classical Thue-Morse sequence. We also establish a variant of van der Corput's Difference Theorem that is connected to unitary operators with continuous spectrum. Lastly, we obtain a new characterization of those sequence $(x_n)_{n = 1}^\infty \subseteq [0,1]$ for which $(x_{n+h},x_n)_{n = 1}^\infty$ is uniformly distributed in $[0,1]^2$ for all $h \in \mathbb{N}$.

Dynamical Systems
Convergence Rates for the Generalized Fréchet Mean via the Quadruple Inequality

Christof Schötz

2023-06-14

Abstract:For sets $\mathcal Q$ and $\mathcal Y$, the generalized Fréchet mean $m \in \mathcal Q$ of a random variable $Y$, which has values in $\mathcal Y$, is any minimizer of $q\mapsto \mathbb E[\mathfrak c(q,Y)]$, where $\mathfrak c \colon \mathcal Q \times \mathcal Y \to \mathbb R$ is a cost function. There are little restrictions to $\mathcal Q$ and $\mathcal Y$. In particular, $\mathcal Q$ can be a non-Euclidean metric space. We provide convergence rates for the empirical generalized Fréchet mean. Conditions for rates in probability and rates in expectation are given. In contrast to previous results on Fréchet means, we do not require a finite diameter of the $\mathcal Q$ or $\mathcal Y$. Instead, we assume an inequality, which we call quadruple inequality. It generalizes an otherwise common Lipschitz condition on the cost function. This quadruple inequality is known to hold in Hadamard spaces. We show that it also holds in a suitable way for certain powers of a Hadamard-metric.

Statistics Theory
The Khinchin-Kahane and Levy inequalities for abelian metric groups, and transfer from normed (abelian semi)groups to Banach spaces

Apoorva Khare,Bala Rajaratnam

DOI: https://doi.org/10.1016/j.jmaa.2023.127545

2023-06-23

Abstract:The Khinchin-Kahane inequality is a fundamental result in the probability literature, with the most general version to date holding in Banach spaces. Motivated by modern settings and applications, we generalize this inequality to arbitrary metric groups which are abelian. If instead of abelian one assumes the group's metric to be a norm (i.e., $\mathbb{Z}_{>0}$-homogeneous), then we explain how the inequality improves to the same one as in Banach spaces. This occurs via a "transfer principle" that helps carry over questions involving normed metric groups and abelian normed semigroups into the Banach space framework. This principle also extends the notion of the expectation to random variables with values in arbitrary abelian normed metric semigroups $\mathscr{G}$. We provide additional applications, including studying weakly $\ell_p$ $\mathscr{G}$-valued sequences and related Rademacher series. On a related note, we also formulate a "general" Levy inequality, with two features: (i) It subsumes several known variants in the Banach space literature; and (ii) We show the inequality in the minimal framework required to state it: abelian metric groups.

Probability,Group Theory

An extension of McDiarmid's inequality

On Bernstein Type Inequalities for Stochastic Integrals of Multivariate Point Processes

Distribution-dependent McDiarmid-type Inequalities for Functions of Unbounded Interaction.

Distribution-dependent Concentration Inequalities for Tighter Generalization Bounds

Exponential inequalities in probability spaces revisited

Concentration Inequalities for General Functions of Heavy-Tailed Random Variables

Asymmetric Doob Inequalities in Continuous Time

On a version of a multivariate integration by parts formula for Lebesgue integrals

A uniform Dvoretzky-Kiefer-Wolfowitz inequality

Bernstein-type and Bennett-type inequalities for unbounded matrix martingales

Concentration inequalities for locally small increments of compound empirical processes with applications to solutions of compound and risk averse stochastical programming

Hoeffding's inequality for supermartingales

Concentration Inequalities for Sums of Markov Dependent Random Matrices

A Doob-type maximal inequality and its applications to various stochastic processes

A short proof of the Dvoretzky--Kiefer--Wolfowitz--Massart inequality

A Matrix Bernstein Inequality for Strong Rayleigh Distributions

Stability for the Brunn-Minkowski and Riesz Rearrangement Inequalities, with Applications to Gaussian Concentration and Finite Range Non-local Isoperimetry

The extended Ville's inequality for nonintegrable nonnegative supermartingales

A generalization of van der Corput's Difference Theorem

Convergence Rates for the Generalized Fréchet Mean via the Quadruple Inequality

The Khinchin-Kahane and Levy inequalities for abelian metric groups, and transfer from normed (abelian semi)groups to Banach spaces