Anytime-Valid Tests of Group Invariance through Conformal Prediction

Tyron Lardy,Muriel Felipe Pérez-Ortiz

2024-05-23

Abstract:We develop anytime-valid tests of invariance under the action of compact groups. The resulting test statistics are optimal in a logarithmic-growth sense. We apply our method to extend recent anytime-valid tests of independence and to construct tests of normality.

Methodology,Statistics Theory

What problem does this paper attempt to address?

E-Statistics, Group Invariance and Anytime Valid Testing

Muriel Felipe Pérez-Ortiz,Tyron Lardy,Rianne de Heide,Peter Grünwald

2023-10-17

Abstract:We study worst-case-growth-rate-optimal (GROW) e-statistics for hypothesis testing between two group models. It is known that under a mild condition on the action of the underlying group G on the data, there exists a maximally invariant statistic. We show that among all e-statistics, invariant or not, the likelihood ratio of the maximally invariant statistic is GROW, both in the absolute and in the relative sense, and that an anytime-valid test can be based on it. The GROW e-statistic is equal to a Bayes factor with a right Haar prior on G. Our treatment avoids nonuniqueness issues that sometimes arise for such priors in Bayesian contexts. A crucial assumption on the group G is its amenability, a well-known group-theoretical condition, which holds, for instance, in scale-location families. Our results also apply to finite-dimensional linear regression.

Statistics Theory,Methodology
Post-hoc and Anytime Valid Permutation and Group Invariance Testing

Nick W. Koning

2024-03-05

Abstract:We study post-hoc ($e$-value-based) and post-hoc anytime valid inference for testing exchangeability and general group invariance. Our methods satisfy a generalized Type I error control that permits a data-dependent selection of both the number of observations $n$ and the significance level $\alpha$. We derive a simple analytical expression for all exact post-hoc valid $p$-values for group invariance, which allows for a flexible plug-in of the test statistic. For post-hoc anytime validity, we derive sequential $p$-processes by multiplying post-hoc $p$-values. In sequential testing, it is key to specify how the number of observations may depend on the data. We propose two approaches, and show how they nest existing efforts. To construct good post-hoc $p$-values, we develop the theory of likelihood ratios for group invariance, and generalize existing optimality results. These likelihood ratios turn out to exist in different flavors depending on which space we specify our alternative. We illustrate our methods by testing against a Gaussian location shift, which yields an improved optimality result for the $t$-test when testing sphericity, connections to the softmax function when testing exchangeability, and an improved method for testing sign-symmetry.

Statistics Theory,Methodology
Consistency of invariance-based randomization tests

Edgar Dobriban

DOI: https://doi.org/10.48550/arXiv.2104.12260

2021-12-20

Abstract:Invariance-based randomization tests -- such as permutation tests, rotation tests, or sign changes -- are an important and widely used class of statistical methods. They allow drawing inferences under weak assumptions on the data distribution. Most work focuses on their type I error control properties, while their consistency properties are much less understood. We develop a general framework and a set of results on the consistency of invariance-based randomization tests in signal-plus-noise models. Our framework is grounded in the deep mathematical area of representation theory. We allow the transforms to be general compact topological groups, such as rotation groups, acting by general linear group representations. We study test statistics with a generalized sub-additivity property. We apply our framework to a number of fundamental and highly important problems in statistics, including sparse vector detection, testing for low-rank matrices in noise, sparse detection in linear regression, and two-sample testing. Comparing with minimax lower bounds, we find perhaps surprisingly that in some cases, randomization tests detect signals at the minimax optimal rate.

Statistics Theory,Methodology
Asymptotic symmetry and group invariance for randomization

Adam B Kashlak

2023-10-20

Abstract:Symmetry is a cornerstone of much of mathematics, and many probability distributions possess symmetries characterized by their invariance to a collection of group actions. Thus, many mathematical and statistical methods rely on such symmetry holding and ostensibly fail if symmetry is broken. This work considers under what conditions a sequence of probability measures asymptotically gains such symmetry or invariance to a collection of group actions. Considering the many symmetries of the Gaussian distribution, this work effectively proposes a non-parametric type of central limit theorem. That is, a Lipschitz function of a high dimensional random vector will be asymptotically invariant to the actions of certain compact topological groups. Applications of this include a partial law of the iterated logarithm for uniformly random points in an $\ell_p^n$-ball and an asymptotic equivalence between classical parametric statistical tests and their randomization counterparts even when invariance assumptions are violated.

Statistics Theory,Probability
Group conditional validity via multi-group learning

Samuel Deng,Navid Ardeshir,Daniel Hsu

DOI: https://doi.org/10.48550/arXiv.2303.03995

IF: 5.414

2023-03-07

Machine Learning

Abstract:We consider the problem of distribution-free conformal prediction and the criterion of group conditional validity. This criterion is motivated by many practical scenarios including hidden stratification and group fairness. Existing methods achieve such guarantees under either restrictive grouping structure or distributional assumptions, or they are overly-conservative under heteroskedastic noise. We propose a simple reduction to the problem of achieving validity guarantees for individual populations by leveraging algorithms for a problem called multi-group learning. This allows us to port theoretical guarantees from multi-group learning to obtain obtain sample complexity guarantees for conformal prediction. We also provide a new algorithm for multi-group learning for groups with hierarchical structure. Using this algorithm in our reduction leads to improved sample complexity guarantees with a simpler predictor structure.
Transductive conformal inference with adaptive scores

Ulysse Gazin,Gilles Blanchard,Etienne Roquain

DOI: https://doi.org/10.48550/arXiv.2310.18108

2023-10-27

Methodology

Abstract:Conformal inference is a fundamental and versatile tool that provides distribution-free guarantees for many machine learning tasks. We consider the transductive setting, where decisions are made on a test sample of $m$ new points, giving rise to $m$ conformal $p$-values. {While classical results only concern their marginal distribution, we show that their joint distribution follows a P\'olya urn model, and establish a concentration inequality for their empirical distribution function.} The results hold for arbitrary exchangeable scores, including {\it adaptive} ones that can use the covariates of the test+calibration samples at training stage for increased accuracy. We demonstrate the usefulness of these theoretical results through uniform, in-probability guarantees for two machine learning tasks of current interest: interval prediction for transductive transfer learning and novelty detection based on two-class classification.
Conformal Inference for Invariant Risk Minimization

Wenlu Tang,Zicheng Liu

DOI: https://doi.org/10.48550/arXiv.2305.12686

2023-05-22

Abstract:The application of machine learning models can be significantly impeded by the occurrence of distributional shifts, as the assumption of homogeneity between the population of training and testing samples in machine learning and statistics may not be feasible in practical situations. One way to tackle this problem is to use invariant learning, such as invariant risk minimization (IRM), to acquire an invariant representation that aids in generalization with distributional shifts. This paper develops methods for obtaining distribution-free prediction regions to describe uncertainty estimates for invariant representations, accounting for the distribution shifts of data from different environments. Our approach involves a weighted conformity score that adapts to the specific environment in which the test sample is situated. We construct an adaptive conformal interval using the weighted conformity score and prove its conditional average under certain conditions. To demonstrate the effectiveness of our approach, we conduct several numerical experiments, including simulation studies and a practical example using real-world data.

Machine Learning
Bi-invariant Two-Sample Tests in Lie Groups for Shape Analysis

Martin Hanik,Hans-Christian Hege,Christoph von Tycowicz

DOI: https://doi.org/10.1007/978-3-030-61056-2_4

2020-08-27

Abstract:We propose generalizations of the Hotelling's $T^2$ statistic and the Bhattacharayya distance for data taking values in Lie groups. A key feature of the derived measures is that they are compatible with the group structure even for manifolds that do not admit any bi-invariant metric. This property, e.g., assures analysis that does not depend on the reference shape, thus, preventing bias due to arbitrary choices thereof. Furthermore, the generalizations agree with the common definitions for the special case of flat vector spaces guaranteeing consistency. Employing a permutation test setup, we further obtain nonparametric, two-sample testing procedures that themselves are bi-invariant and consistent. We validate our method in group tests revealing significant differences in hippocampal shape between individuals with mild cognitive impairment and normal controls.

Statistics Theory
Probabilistic Conformal Prediction with Approximate Conditional Validity

Vincent Plassier,Alexander Fishkov,Mohsen Guizani,Maxim Panov,Eric Moulines

2024-10-09

Abstract:We develop a new method for generating prediction sets that combines the flexibility of conformal methods with an estimate of the conditional distribution $P_{Y \mid X}$. Existing methods, such as conformalized quantile regression and probabilistic conformal prediction, usually provide only a marginal coverage guarantee. In contrast, our approach extends these frameworks to achieve approximately conditional coverage, which is crucial for many practical applications. Our prediction sets adapt to the behavior of the predictive distribution, making them effective even under high heteroscedasticity. While exact conditional guarantees are infeasible without assumptions on the underlying data distribution, we derive non-asymptotic bounds that depend on the total variation distance of the conditional distribution and its estimate. Using extensive simulations, we show that our method consistently outperforms existing approaches in terms of conditional coverage, leading to more reliable statistical inference in a variety of applications.

Machine Learning,Probability,Statistics Theory,Methodology
On the Validity of Conformal Prediction for Network Data Under Non-Uniform Sampling

Robert Lunde

2023-07-13

Abstract:We study the properties of conformal prediction for network data under various sampling mechanisms that commonly arise in practice but often result in a non-representative sample of nodes. We interpret these sampling mechanisms as selection rules applied to a superpopulation and study the validity of conformal prediction conditional on an appropriate selection event. We show that the sampled subarray is exchangeable conditional on the selection event if the selection rule satisfies a permutation invariance property and a joint exchangeability condition holds for the superpopulation. Our result implies the finite-sample validity of conformal prediction for certain selection events related to ego networks and snowball sampling. We also show that when data are sampled via a random walk on a graph, a variant of weighted conformal prediction yields asymptotically valid prediction sets for an independently selected node from the population.

Statistics Theory,Methodology,Machine Learning
Asymptotics for conformal inference

Ulysse Gazin

2024-09-18

Abstract:Conformal inference is a versatile tool for building prediction sets in regression or classification. In this paper, we consider the false coverage proportion (FCP) in a transductive setting with a calibration sample of n points and a test sample of m points. We identify the exact, distribution-free, asymptotic distribution of the FCP when both n and m tend to infinity. This shows in particular that FCP control can be achieved by using the well-known Kolmogorov distribution, and puts forward that the asymptotic variance is decreasing in the ratio n/m. We then provide a number of extensions by considering the novelty detection problem, weighted conformal inference and distribution shift between the calibration sample and the test sample. In particular, our asymptotical results allow to accurately quantify the asymptotical behavior of the errors when weighted conformal inference is used.

Statistics Theory
Localized Conformal Prediction: A Generalized Inference Framework for Conformal Prediction

Leying Guan

DOI: https://doi.org/10.48550/arXiv.2106.08460

2022-03-01

Abstract:We propose a new inference framework called localized conformal prediction. It generalizes the framework of conformal prediction by offering a single-test-sample adaptive construction that emphasizes a local region around this test sample, and can be combined with different conformal score constructions. The proposed framework enjoys an assumption-free finite sample marginal coverage guarantee, and it also offers additional local coverage guarantees under suitable assumptions. We demonstrate how to change from conformal prediction to localized conformal prediction using several conformal scores, and we illustrate a potential gain via numerical examples.

Statistics Theory,Methodology
Approximation to Object Conditional Validity with Inductive Conformal Predictors

Anthony Bellotti

DOI: https://doi.org/10.48550/arXiv.2102.07436

2021-03-02

Abstract:Conformal predictors are machine learning algorithms that output prediction sets that have a guarantee of marginal validity for finite samples with minimal distributional assumptions. This is a property that makes conformal predictors useful for machine learning tasks where we require reliable predictions. It would also be desirable to achieve conditional validity in the same setting, in the sense that validity of the prediction intervals remains valid regardless of conditioning on any particular property of the object of the prediction. Unfortunately, it has been shown that such conditional validity is impossible to guarantee for non-trivial prediction problems for finite samples. In this article, instead of trying to achieve a strong conditional validity result, the weaker goal of achieving an approximation to conditional validity is considered. A new algorithm is introduced to do this by iteratively adjusting a conformity measure to deviations from object conditional validity measured in the training data. Along with some theoretical results, experimental results are provided for three data sets that demonstrate (1) in real world machine learning tasks, lack of conditional validity is a measurable problem and (2) that the proposed algorithm is effective at alleviating this problem.

Machine Learning
Exact and Robust Conformal Inference Methods for Predictive Machine Learning With Dependent Data

Victor Chernozhukov,Kaspar Wuthrich,Yinchu Zhu

DOI: https://doi.org/10.48550/arXiv.1802.06300

2018-07-12

Abstract:We extend conformal inference to general settings that allow for time series data. Our proposal is developed as a randomization method and accounts for potential serial dependence by including block structures in the permutation scheme. As a result, the proposed method retains the exact, model-free validity when the data are i.i.d. or more generally exchangeable, similar to usual conformal inference methods. When exchangeability fails, as is the case for common time series data, the proposed approach is approximately valid under weak assumptions on the conformity score.

Machine Learning
Nothing Conformal about Adaptive Conformal Inference

Johan Hallberg Szabadváry

2024-09-26

Abstract:Conformal prediction is a widely-used framework for distribution-free uncertainty quantification, which generates valid prediction sets at a user-defined significance level. However, this framework relies on the assumption that the data-generating distribution is exchangeable, a condition that is frequently violated in time-series and other structured data. In such cases, the validity guarantees of conformal prediction break down. Adaptive conformal inference (ACI) has been proposed as a solution for non-exchangeable data by dynamically adjusting the significance level to retain at least finite sample guarantees on the marginal coverage error rate. This paper demonstrates that, despite its name, ACI does not strictly require the use of conformal predictors. Instead, it can operate effectively with the more general concept of a confidence predictor, which is often computationally simpler. The key requirement is that larger significance levels correspond to smaller prediction sets, a property known as nested prediction sets. Through experiments on synthetic and real-world data, we investigate whether ACI with conformal predictors offers advantages over confidence predictors. Our results indicate that confidence predictors can perform just as well, and sometimes better than conformal predictors in some cases, although further empirical studies are needed to determine when one approach may be preferable.

Machine Learning
Beyond Conformal Predictors: Adaptive Conformal Inference with Confidence Predictors

Johan Hallberg Szabadváry

2024-10-25

Abstract:Conformal prediction (CP) is a robust framework for distribution-free uncertainty quantification, but it requires exchangeable data to ensure valid prediction sets at a user-specified significance level. When this assumption is violated, as in time-series or other structured data, the validity guarantees of CP no longer hold. Adaptive conformal inference (ACI) was introduced to address this limitation by adjusting the significance level dynamically, ensuring finite-sample coverage guarantees even for non-exchangeable data. In this paper, we show that ACI does not require the use of conformal predictors; instead, it can be implemented with the more general confidence predictors, which are computationally simpler and still maintain the crucial property of nested prediction sets. Through experiments on synthetic and real-world data, we demonstrate that confidence predictors can perform comparably to, or even better than, conformal predictors, particularly in terms of computational efficiency. These findings suggest that confidence predictors represent a viable and efficient alternative to conformal predictors in non-exchangeable data settings, although further studies are needed to identify when one method is superior.

Machine Learning
Conformal Validity Guarantees Exist for Any Data Distribution (and How to Find Them)

Drew Prinster,Samuel Stanton,Anqi Liu,Suchi Saria

2024-06-05

Abstract:As artificial intelligence (AI) / machine learning (ML) gain widespread adoption, practitioners are increasingly seeking means to quantify and control the risk these systems incur. This challenge is especially salient when such systems have autonomy to collect their own data, such as in black-box optimization and active learning, where their actions induce sequential feedback-loop shifts in the data distribution. Conformal prediction is a promising approach to uncertainty and risk quantification, but prior variants' validity guarantees have assumed some form of ``quasi-exchangeability'' on the data distribution, thereby excluding many types of sequential shifts. In this paper we prove that conformal prediction can theoretically be extended to \textit{any} joint data distribution, not just exchangeable or quasi-exchangeable ones. Although the most general case is exceedingly impractical to compute, for concrete practical applications we outline a procedure for deriving specific conformal algorithms for any data distribution, and we use this procedure to derive tractable algorithms for a series of AI/ML-agent-induced covariate shifts. We evaluate the proposed algorithms empirically on synthetic black-box optimization and active learning tasks.

Machine Learning,Artificial Intelligence
Conformal Prediction Under Covariate Shift

Ryan J. Tibshirani,Rina Foygel Barber,Emmanuel J. Candes,Aaditya Ramdas

DOI: https://doi.org/10.48550/arXiv.1904.06019

2020-07-07

Abstract:We extend conformal prediction methodology beyond the case of exchangeable data. In particular, we show that a weighted version of conformal prediction can be used to compute distribution-free prediction intervals for problems in which the test and training covariate distributions differ, but the likelihood ratio between these two distributions is known---or, in practice, can be estimated accurately with access to a large set of unlabeled data (test covariate points). Our weighted extension of conformal prediction also applies more generally, to settings in which the data satisfies a certain weighted notion of exchangeability. We discuss other potential applications of our new conformal methodology, including latent variable and missing data problems.

Methodology
Conformal prediction with local weights: randomization enables local guarantees

Rohan Hore,Rina Foygel Barber

DOI: https://doi.org/10.48550/arXiv.2310.07850

2024-10-26

Abstract:In this work, we consider the problem of building distribution-free prediction intervals with finite-sample conditional coverage guarantees. Conformal prediction (CP) is an increasingly popular framework for building prediction intervals with distribution-free guarantees, but these guarantees only ensure marginal coverage: the probability of coverage is averaged over a random draw of both the training and test data, meaning that there might be substantial undercoverage within certain subpopulations. Instead, ideally, we would want to have local coverage guarantees that hold for each possible value of the test point's features. While the impossibility of achieving pointwise local coverage is well established in the literature, many variants of conformal prediction algorithm show favorable local coverage properties empirically. Relaxing the definition of local coverage can allow for a theoretical understanding of this empirical phenomenon. We aim to bridge this gap between theoretical validation and empirical performance by proving achievable and interpretable guarantees for a relaxed notion of local coverage. Building on the localized CP method of Guan (2023) and the weighted CP framework of Tibshirani et al. (2019), we propose a new method, randomly-localized conformal prediction (RLCP), which returns prediction intervals that are not only marginally valid but also achieve a relaxed local coverage guarantee and guarantees under covariate shift. Through a series of simulations and real data experiments, we validate these coverage guarantees of RLCP while comparing it with the other local conformal prediction methods.

Methodology
Robust Conformal Prediction Using Privileged Information

Shai Feldman,Yaniv Romano

2024-09-27

Abstract:We develop a method to generate prediction sets with a guaranteed coverage rate that is robust to corruptions in the training data, such as missing or noisy variables. Our approach builds on conformal prediction, a powerful framework to construct prediction sets that are valid under the i.i.d assumption. Importantly, naively applying conformal prediction does not provide reliable predictions in this setting, due to the distribution shift induced by the corruptions. To account for the distribution shift, we assume access to privileged information (PI). The PI is formulated as additional features that explain the distribution shift, however, they are only available during training and absent at test time. We approach this problem by introducing a novel generalization of weighted conformal prediction and support our method with theoretical coverage guarantees. Empirical experiments on both real and synthetic datasets indicate that our approach achieves a valid coverage rate and constructs more informative predictions compared to existing methods, which are not supported by theoretical guarantees.

Machine Learning

Anytime-Valid Tests of Group Invariance through Conformal Prediction

E-Statistics, Group Invariance and Anytime Valid Testing

Post-hoc and Anytime Valid Permutation and Group Invariance Testing

Consistency of invariance-based randomization tests

Asymptotic symmetry and group invariance for randomization

Group conditional validity via multi-group learning

Transductive conformal inference with adaptive scores

Conformal Inference for Invariant Risk Minimization

Bi-invariant Two-Sample Tests in Lie Groups for Shape Analysis

Probabilistic Conformal Prediction with Approximate Conditional Validity

On the Validity of Conformal Prediction for Network Data Under Non-Uniform Sampling

Asymptotics for conformal inference

Localized Conformal Prediction: A Generalized Inference Framework for Conformal Prediction

Approximation to Object Conditional Validity with Inductive Conformal Predictors

Exact and Robust Conformal Inference Methods for Predictive Machine Learning With Dependent Data

Nothing Conformal about Adaptive Conformal Inference

Beyond Conformal Predictors: Adaptive Conformal Inference with Confidence Predictors

Conformal Validity Guarantees Exist for Any Data Distribution (and How to Find Them)

Conformal Prediction Under Covariate Shift

Conformal prediction with local weights: randomization enables local guarantees

Robust Conformal Prediction Using Privileged Information