Introduction to Concentration Inequalities

Kumar Abhishek,Sneha Maheshwari,Sujit Gujar

DOI: https://doi.org/10.48550/arXiv.1910.02884

2019-10-04

Abstract:In this report, we aim to exemplify concentration inequalities and provide easy to understand proofs for it. Our focus is on the inequalities which are helpful in the design and analysis of machine learning algorithms.

Probability,Machine Learning,Statistics Theory

What problem does this paper attempt to address?

Concentration Inequalities for Statistical Inference

Huiming Zhang,Song Xi Chen

DOI: https://doi.org/10.4208/cmr.2020-0041

2021-01-01

Communications in Mathematical Research

Abstract:This paper gives a review of concentration inequalities which are widely employed in non-asymptotical analyses of mathematical statistics in a wide range of settings, from distribution-free to distribution-dependent, from sub-Gaussian to sub-exponential, sub-Gamma, and sub-Weibull random variables, and from the mean to the maximum concentration. This review provides results in these settings with some fresh new results. Given the increasing popularity of high-dimensional data and inference, results in the context of high-dimensional linear and Poisson regressions are also provided. We aim to illustrate the concentration inequalities with known constants and to improve existing bounds with sharper constants.
The Iyengar Inequality, Revisited

X. H. Wang,C. C. Xie,S. J. Yang

DOI: https://doi.org/10.1007/s10543-007-0153-9

2007-01-01

Abstract:Detailed analysis shows that the famous Iyengar inequality actually says that the Trapezoidal formula is a central algorithm for approximating integrals over an appropriate interval for the class of functions whose derivatives are bounded by a positive number K in L ∞ -sense. The inherent nonlinearity from central algorithms reflects the importance of the Iyengar inequality and thus makes familiar linear methods malfunction when one tries to generalize it. It is shown that the generalization depends on a nonlinear system of equations satisfied by a set of free nodes of a perfect spline. Explicit constructions are obtained in the spirit of the Iyengar inequality for the class of functions whose r th ( r ≤4) derivatives are bounded by a positive number K in L ∞ -sense because a closed solution to the nonlinear system is only available for r ≤4. Connections with computational mathematics, especially with best interpolation and best quadrature, are discussed. Numerical experiments are also included.
The Iyengar Type Inequalities with Exact Estimations and the Chebyshev Central Algorithms of Integrals

Xing Hua Wang,Shi Jun Yang

DOI: https://doi.org/10.1007/s10114-005-0578-5

2005-01-01

Acta Mathematica Sinica English Series

Abstract:In this paper, both low order and high order extensions of the Iyengar type inequality are obtained. Such extensions are the best possible in the same sense as that of the Iyengar inequality. Furthermore, the Chebyshev central algorithms of integrals for some function classes and some related problems are also considered and investigated.
A Note on Concentration Inequalities for U-Statistics

Yannik Pitcan

DOI: https://doi.org/10.48550/arXiv.1712.06160

2019-03-15

Abstract:The aim of this paper is to discuss various concentration inequalities for U-statistics and most recent results. A special focus will be on providing proofs for bounds on the U-statistics using classical concentration inequalities, which, although the results well known, the proofs are not found in the literature.

Statistics Theory
Constructive approaches to concentration inequalities with independent random variables

Celine Moucer,Adrien Taylor,Francis Bach

2024-08-29

Abstract:Concentration inequalities, a major tool in probability theory, quantify how much a random variable deviates from a certain quantity. This paper proposes a systematic convex optimization approach to studying and generating concentration inequalities with independent random variables. Specifically, we extend the generalized problem of moments to independent random variables. We first introduce a variational approach that extends classical moment-generating functions, focusing particularly on first-order moment conditions. Second, we develop a polynomial approach, based on a hierarchy of sum-of-square approximations, to extend these techniques to higher-moment conditions. Building on these advancements, we refine Hoeffding's, Bennett's and Bernstein's inequalities, providing improved worst-case guarantees compared to existing results.

Probability,Optimization and Control
Concentration Inequalities from Likelihood Ratio Method

Xinjia Chen

DOI: https://doi.org/10.48550/arXiv.1409.6276

2014-09-02

Abstract:We explore the applications of our previously established likelihood-ratio method for deriving concentration inequalities for a wide variety of univariate and multivariate distributions. New concentration inequalities for various distributions are developed without the idea of minimizing moment generating functions.

Statistics Theory,Probability
Concentration of Measure Inequalities in Information Theory, Communications, and Coding

Maxim Raginsky,Igal Sason

DOI: https://doi.org/10.1561/0100000064

2013-01-01

Foundations and Trends® in Communications and Information Theory

Abstract:Concentration inequalities have been the subject of exciting developments during the last two decades, and have been intensively studied and used as a powerful tool in various areas. These include convex geometry, functional analysis, statistical physics, mathematical statistics, pure and applied probability theory (e.g., concentration of measure phenomena in random graphs, random matrices, and percolation), information theory, theoretical computer science, learning theory, and dynamical systems. This monograph focuses on some of the key modern mathematical tools that are used for the derivation of concentration inequalities, on their links to information theory, and on their various applications to communications and coding. In addition to being a survey, this monograph also includes various new recent results derived by the authors. The first part of the monograph introduces classical concentration inequalities for martingales, as well as some recent refinements and extensions. The power and versatility of the martingale approach is exemplified in the context of codes defined on graphs and iterative decoding algorithms, as well as codes for wireless communication. The second part of the monograph introduces the entropy method, an information-theoretic technique for deriving concentration inequalities for functions of many independent random variables. The basic ingredients of the entropy method are discussed first in conjunction with the closely related topic of logarithmic Sobolev inequalities, which are typical of the so-called functional approach to studying the concentration of measure phenomenon. The discussion on logarithmic Sobolev inequalities is complemented by a related viewpoint based on probability in metric spaces. This viewpoint centers around the so-called transportation-cost inequalities, whose roots are in information theory. Some representative results on concentration for dependent random variables are briefly summarized, with emphasis on their connections to the entropy method. Finally, we discuss several applications of the entropy method and related information-theoretic tools to problems in communications and coding. These include strong converses, empirical distributions of good channel codes with non-vanishing error probability, and an information-theoretic converse for concentration of measure.
From Poincaré Inequalities to Nonlinear Matrix Concentration

De Huang,Joel A. Tropp

DOI: https://doi.org/10.48550/arXiv.2006.16561

2021-01-07

Abstract:This paper deduces exponential matrix concentration from a Poincaré inequality via a short, conceptual argument. Among other examples, this theory applies to matrix-valued functions of a uniformly log-concave random vector. The proof relies on the subadditivity of Poincaré inequalities and a chain rule inequality for the trace of the matrix Dirichlet form. It also uses a symmetrization technique to avoid difficulties associated with a direct extension of the classic scalar argument.

Probability
Concentration Inequalities for Bounded Random Vectors

Xinjia Chen

DOI: https://doi.org/10.48550/arXiv.1309.0003

2013-08-31

Abstract:We derive simple concentration inequalities for bounded random vectors, which generalize Hoeffding's inequalities for bounded scalar random variables. As applications, we apply the general results to multinomial and Dirichlet distributions to obtain multivariate concentration inequalities.

Probability,Machine Learning,Statistics Theory
Deriving Matrix Concentration Inequalities from Kernel Couplings

Daniel Paulin,Lester Mackey,Joel A. Tropp

DOI: https://doi.org/10.48550/arXiv.1305.0612

2013-05-03

Abstract:This paper derives exponential tail bounds and polynomial moment inequalities for the spectral norm deviation of a random matrix from its mean value. The argument depends on a matrix extension of Stein's method of exchangeable pairs for concentration of measure, as introduced by Chatterjee. Recent work of Mackey et al. uses these techniques to analyze random matrices with additive structure, while the enhancements in this paper cover a wider class of matrix-valued random elements. In particular, these ideas lead to a bounded differences inequality that applies to random matrices constructed from weakly dependent random variables. The proofs require novel trace inequalities that may be of independent interest.

Probability,Functional Analysis
Concentration-Bound Analysis for Probabilistic Programs and Probabilistic Recurrence Relations

Jinyi Wang,Yican Sun,Hongfei Fu,Mingzhang Huang,Amir Kafshdar Goharshady,Krishnendu Chatterjee

DOI: https://doi.org/10.48550/arXiv.2008.00425

2020-08-12

Abstract:Analyzing probabilistic programs and randomized algorithms are classical problems in computer science. The first basic problem in the analysis of stochastic processes is to consider the expectation or mean, and another basic problem is to consider concentration bounds, i.e. showing that large deviations from the mean have small probability. Similarly, in the context of probabilistic programs and randomized algorithms, the analysis of expected termination time/running time and their concentration bounds are fundamental <a class="link-external link-http" href="http://problems.In" rel="external noopener nofollow">this http URL</a> this work, we focus on concentration bounds for probabilistic programs and probabilistic recurrences of randomized algorithms. For probabilistic programs, the basic technique to achieve concentration bounds is to consider martingales and apply the classical Azuma's inequality. For probabilistic recurrences of randomized algorithms, Karp's classical "cookbook" method, which is similar to the master theorem for recurrences, is the standard approach to obtain concentration bounds. In this work, we propose a novel approach for deriving concentration bounds for probabilistic programs and probabilistic recurrence relations through the synthesis of exponential supermartingales. For probabilistic programs, we present algorithms for synthesis of such supermartingales in several cases. We also show that our approach can derive better concentration bounds than simply applying the classical Azuma's inequality over various probabilistic programs considered in the literature. For probabilistic recurrences, our approach can derive tighter bounds than the Karp's well-established methods on classical algorithms. Moreover, we show that our approach could derive bounds comparable to the optimal bound for quicksort, proposed by McDiarmid and Hayward. We also present a prototype implementation that can automatically infer these bounds

Programming Languages,Data Structures and Algorithms
Efficient Concentration with Gaussian Approximation

Morgane Austern,Lester Mackey

2024-05-22

Abstract:Concentration inequalities for the sample mean, like those due to Bernstein and Hoeffding, are valid for any sample size but overly conservative, yielding confidence intervals that are unnecessarily wide. The central limit theorem (CLT) provides asymptotic confidence intervals with optimal width, but these are invalid for all sample sizes. To resolve this tension, we develop new computable concentration inequalities with asymptotically optimal size, finite-sample validity, and sub-Gaussian decay. These bounds enable the construction of efficient confidence intervals with correct coverage for any sample size and efficient empirical Berry-Esseen bounds that require no prior knowledge of the population variance. We derive our inequalities by tightly bounding non-uniform Kolmogorov and Wasserstein distances to a Gaussian using zero-bias couplings and Stein's method of exchangeable pairs.

Probability,Statistics Theory
Equivalence of concentration inequalities for linear and non-linear functions

Timothy John Sullivan,Houman Owhadi

DOI: https://doi.org/10.48550/arXiv.1009.4913

2010-09-25

Abstract:We consider a random variable $X$ that takes values in a (possibly infinite-dimensional) topological vector space $\mathcal{X}$. We show that, with respect to an appropriate "normal distance" on $\mathcal{X}$, concentration inequalities for linear and non-linear functions of $X$ are equivalent. This normal distance corresponds naturally to the concentration rate in classical concentration results such as Gaussian concentration and concentration on the Euclidean and Hamming cubes. Under suitable assumptions on the roundness of the sets of interest, the concentration inequalities so obtained are asymptotically optimal in the high-dimensional limit.

Probability
Concentration Inequalities and Laws of Large Numbers under Epistemic Irrelevance

Fabio Gagliardi Cozman

DOI: https://doi.org/10.48550/arXiv.0810.2821

2008-10-15

Probability

Abstract:This paper presents concentration inequalities and laws of large numbers under weak assumptions of irrelevance, expressed through lower and upper expectations. The results are variants and extensions of De Cooman and Miranda's recent inequalities and laws of large numbers. The proofs indicate connections between concepts of irrelevance for lower/upper expectations and the standard theory of martingales.
Causal inference targeting a concentration index for studies of health inequalities

Mohammad Ghasempour,Xavier de Luna,Per E. Gustafsson

2024-10-11

Abstract:A concentration index, a standardized covariance between a health variable and relative income ranks, is often used to quantify income-related health inequalities. There is a lack of formal approach to study the effect of an exposure, e.g., education, on such measures of inequality. In this paper we contribute by filling this gap and developing the necessary theory and method. Thus, we define a counterfactual concentration index for different levels of an exposure. We give conditions for their identification, and then deduce their efficient influence function. This allows us to propose estimators, which are regular asymptotic linear under certain conditions. In particular, these estimators are $\sqrt n$-consistent and asymptotically normal, as well as locally efficient. The implementation of the estimators is based on the fit of several nuisance functions. The estimators proposed have rate robustness properties allowing for convergence rates slower than $\sqrt{n}$-rate for some of the nuisance function fits. The relevance of the asymptotic results for finite samples is studied with simulation experiments. We also present a case study of the effect of education on income-related health inequalities for a Swedish cohort.

Methodology,Statistics Theory
Designing Equitable Algorithms

Alex Chohlas-Wood,Madison Coots,Sharad Goel,Julian Nyarko

DOI: https://doi.org/10.48550/arXiv.2302.09157

2023-02-18

Abstract:Predictive algorithms are now used to help distribute a large share of our society's resources and sanctions, such as healthcare, loans, criminal detentions, and tax audits. Under the right circumstances, these algorithms can improve the efficiency and equity of decision-making. At the same time, there is a danger that the algorithms themselves could entrench and exacerbate disparities, particularly along racial, ethnic, and gender lines. To help ensure their fairness, many researchers suggest that algorithms be subject to at least one of three constraints: (1) no use of legally protected features, such as race, ethnicity, and gender; (2) equal rates of "positive" decisions across groups; and (3) equal error rates across groups. Here we show that these constraints, while intuitively appealing, often worsen outcomes for individuals in marginalized groups, and can even leave all groups worse off. The inherent trade-off we identify between formal fairness constraints and welfare improvements -- particularly for the marginalized -- highlights the need for a more robust discussion on what it means for an algorithm to be "fair". We illustrate these ideas with examples from healthcare and the criminal-legal system, and make several proposals to help practitioners design more equitable algorithms.

Machine Learning,Computers and Society
Probabilistic verification of fairness properties via concentration

Osbert Bastani,Xin Zhang,Armando Solar-Lezama

DOI: https://doi.org/10.1145/3360544

2019-10-10

Proceedings of the ACM on Programming Languages

Abstract:As machine learning systems are increasingly used to make real world legal and financial decisions, it is of paramount importance that we develop algorithms to verify that these systems do not discriminate against minorities. We design a scalable algorithm for verifying fairness specifications. Our algorithm obtains strong correctness guarantees based on adaptive concentration inequalities; such inequalities enable our algorithm to adaptively take samples until it has enough data to make a decision. We implement our algorithm in a tool called VeriFair, and show that it scales to large machine learning models, including a deep recurrent neural network that is more than five orders of magnitude larger than the largest previously-verified neural network. While our technique only gives probabilistic guarantees due to the use of random samples, we show that we can choose the probability of error to be extremely small.
Combinatorial anti-concentration inequalities, with applications

JACOB FOX,MATTHEW KWAN,LISA SAUERMANN

DOI: https://doi.org/10.1017/s0305004120000183

2021-06-30

Mathematical Proceedings of the Cambridge Philosophical Society

Abstract:Abstract We prove several different anti-concentration inequalities for functions of independent Bernoulli-distributed random variables. First, motivated by a conjecture of Alon, Hefetz, Krivelevich and Tyomkyn, we prove some “Poisson-type” anti-concentration theorems that give bounds of the form 1/ e + o (1) for the point probabilities of certain polynomials. Second, we prove an anti-concentration inequality for polynomials with nonnegative coefficients which extends the classical Erdős–Littlewood–Offord theorem and improves a theorem of Meka, Nguyen and Vu for polynomials of this type. As an application, we prove some new anti-concentration bounds for subgraph counts in random graphs.

mathematics
Inequalities and Convergence Concepts of Fuzzy and Rough Variables

Baoding Liu

DOI: https://doi.org/10.1023/A:1023491000011

IF: 5.274

2003-01-01

Fuzzy Optimization and Decision Making

Abstract:It is well-known that Markov inequality, Chebyshev inequality, Hölder's inequality, and Minkowski inequality are important and useful results in probability theory. This paper presents the analogous inequalities in fuzzy set theory and rough set theory. In addition, sequence convergence plays an extremely important role in the fundamental theory of mathematics. This paper presents four types of convergence concept of fuzzy/rough sequence: convergence almost surely, convergence in credibility/trust, convergence in mean, and convergence in distribution. Some mathematical properties of those new convergence concepts are also given.
A Bennett Inequality for the Missing Mass

Bahman Yari Saeed Khanloo

DOI: https://doi.org/10.48550/arXiv.1503.06134

IF: 5.414

2015-03-20

Machine Learning

Abstract:Novel concentration inequalities are obtained for the missing mass, i.e. the total probability mass of the outcomes not observed in the sample. We derive distribution-free deviation bounds with sublinear exponents in deviation size for missing mass and improve the results of Berend and Kontorovich (2013) and Yari Saeed Khanloo and Haffari (2015) for small deviations which is the most important case in learning theory.

Introduction to Concentration Inequalities

Concentration Inequalities for Statistical Inference

The Iyengar Inequality, Revisited

The Iyengar Type Inequalities with Exact Estimations and the Chebyshev Central Algorithms of Integrals

A Note on Concentration Inequalities for U-Statistics

Constructive approaches to concentration inequalities with independent random variables

Concentration Inequalities from Likelihood Ratio Method

Concentration of Measure Inequalities in Information Theory, Communications, and Coding

From Poincaré Inequalities to Nonlinear Matrix Concentration

Concentration Inequalities for Bounded Random Vectors

Deriving Matrix Concentration Inequalities from Kernel Couplings

Concentration-Bound Analysis for Probabilistic Programs and Probabilistic Recurrence Relations

Efficient Concentration with Gaussian Approximation

Equivalence of concentration inequalities for linear and non-linear functions

Concentration Inequalities and Laws of Large Numbers under Epistemic Irrelevance

Causal inference targeting a concentration index for studies of health inequalities

Designing Equitable Algorithms

Probabilistic verification of fairness properties via concentration

Combinatorial anti-concentration inequalities, with applications

Inequalities and Convergence Concepts of Fuzzy and Rough Variables

A Bennett Inequality for the Missing Mass