Abstract:In computational social science, researchers often use a pre-trained, black box classifier to estimate the frequency of each class in unlabeled datasets. A variety of prevalence estimation techniques have been developed in the literature, each yielding an unbiased estimate if certain stability assumption holds. This work introduces a framework to rethink the prevalence estimation process as calibrating the classifier outputs against ground truth labels to obtain the joint distribution of a base dataset and then extrapolating to the joint distribution of a target dataset. We call this framework "Calibrate-Extrapolate". It clarifies what stability assumptions must hold for a prevalence estimation technique to yield accurate estimates. In the calibration phase, the techniques assume only a stable calibration curve between a calibration dataset and the full base dataset. This allows for the classifier outputs to be used for disproportionate random sampling, thus improving the efficiency of calibration. In the extrapolation phase, some techniques assume a stable calibration curve while some assume stable class-conditional densities. We discuss the stability assumptions from a causal perspective. By specifying base and target joint distributions, we can generate simulated datasets, as a way to build intuitions about the impacts of assumption violations. This also leads to a better understanding of how the classifier's predictive power affects the accuracy of prevalence estimates: the greater the predictive power, the lower the sensitivity to violations of stability assumptions in the extrapolation phase. We illustrate the framework with an application that estimates the prevalence of toxic comments on news topics over time on Reddit, Twitter/X, and YouTube, using Jigsaw's Perspective API as a black box classifier. Finally, we summarize several practical advice for prevalence estimation.

Bayes, Neyman, and Calibration

Conditional calibration and the sage statistician

Robust Bayesian Inference for Berkson and Classical Measurement Error Models

Calibrating Model-Based Inferences and Decisions

A Bayesian treatment of the German tank problem

Bayesian Projected Calibration of Computer Models

Fully Bayesian Benchmarking of Small Area Estimation Models

Bayesian Adaptive Calibration and Optimal Design

Calibrated Bayes, for Statistics in General, and Missing Data in Particular

Non‐parametric inference on calibration of predicted risks

Bayesian correction for covariate measurement error: a frequentist evaluation and comparison with regression calibration

BayCANN: Streamlining Bayesian Calibration with Artificial Neural Network Metamodeling

Bayesian Uncertainty Estimation Under Complex Sampling

Calibration as estimation

A Bayesian framework for functional calibration of expensive computational models through non-isometric matching

Functional Calibration under Non-Probability Survey Sampling

A Bayesian Adjustment for Multiplicative Measurement Errors for a Calibration Problem with Application to a Stem Cell Study

A puzzle of proportions: Two popular Bayesian tests can yield dramatically different conclusions

Calibrate-Extrapolate: Rethinking Prevalence Estimation with Black Box Classifiers

Multiple bias-calibration for adjusting selection bias of non-probability samples using data integration

A Comparison of Economic Agent-Based Model Calibration Methods