Abstract:In this paper we introduce a kernel-based measure for detecting differences between two conditional distributions. Using the `kernel trick' and nearest-neighbor graphs, we propose a consistent estimate of this measure which can be computed in nearly linear time (for a fixed number of nearest neighbors). Moreover, when the two conditional distributions are the same, the estimate has a Gaussian limit and its asymptotic variance has a simple form that can be easily estimated from the data. The resulting test attains precise asymptotic level and is universally consistent for detecting differences between two conditional distributions. We also provide a resampling based test using our estimate that applies to the conditional goodness-of-fit problem, which controls Type I error in finite samples and is asymptotically consistent with only a finite number of resamples. A method to de-randomize the resampling test is also presented. The proposed methods can be readily applied to a broad range of problems, ranging from classical nonparametric statistics to modern machine learning. Specifically, we explore three applications: testing model calibration, regression curve evaluation, and validation of emulator models in simulation-based inference. We illustrate the superior performance of our method for these tasks, both in simulations as well as on real data. In particular, we apply our method to (1) assess the calibration of neural network models trained on the CIFAR-10 dataset, (2) compare regression functions for wind power generation across two different turbines, and (3) validate emulator models on benchmark examples with intractable posteriors and for generating synthetic `redshift' associated with galaxy images.

A New Equating Method:the Kernel Method of Test Equating

Non-parametric Tests for the Tail Equivalence Via Empirical Likelihood

Kimel: A Kernel Incremental Metalearning Algorithm

Comparison of Outlier Detection Methods in NEAT Design

Efficient and powerful equivalency test on combined mean and variance with application to diagnostic device comparison studies

An Extension of Testlet-Based Equating to the Polytomous Testlet Response Theory Model

Kernel Least-Mean Mixed-Norm Algorithm

Using a Linear Regression Method to Detect Outliers in IRT Common Item Equating

A Comparative Review of Methods for Comparing Means Using Partially Paired Data

The Extended Anderson and Hauck Tests and Sample Size Procedures for Equivalence Assessment in Simple Linear Regressions

A Kernel Method for the Two-Sample Problem

A Kernel-Based Conditional Two-Sample Test Using Nearest Neighbors (with Applications to Calibration, Regression Curves, and Simulation-Based Inference)

Wald tests for variance-adjusted equivalence assessment with normal endpoints

Standardisation-function Kernel Stein Discrepancy: A Unifying View on Kernel Stein Discrepancy Tests for Goodness-of-fit

A Novel Iterative Method for Turbo Equalization

An Efficient Kernel Evaluation Criterion for Multiclass Classification

A Comparative Study of Two Kernel Ideas for Nonlinear Feature Extraction

Nonparametric Score Estimators

Equivalence Test in Multi-dimensional Space with Applications in A/B Testing

Generalized kernel two-sample tests

A Comparison Study of K-gap Metric Calculation Based on Different Data-driven Stable Kernel Representation Methods