Abstract:Accurately predicting the probabilities of user feedback, such as clicks and conversions, is critical for advertisement ranking and bidding. However, there often exist unwanted mismatches between predicted probabilities and true likelihoods due to the rapid shift of data distributions and intrinsic model biases. Calibration aims to address this issue by post-processing model predictions, and field-aware calibration can adjust model output on different feature field values to satisfy fine-grained advertising demands. Unfortunately, the observed samples corresponding to certain field values can be seriously limited to make confident calibrations, which may yield bias amplification and online disturbance. In this paper, we propose a confidence-aware multi-field calibration method, which adaptively adjusts the calibration intensity based on confidence levels derived from sample statistics. It also utilizes multiple fields for joint model calibration according to their importance to mitigate the impact of data sparsity on a single field. Extensive offline and online experiments show the superiority of our method in boosting advertising performance and reducing prediction deviations.

What problem does this paper attempt to address?

The problem that this paper attempts to solve is the deviation between the predicted click - through rate (CTR) and conversion rate (CVR) probabilities by the model and the actual probabilities in the online advertising system. This deviation may lead to poor advertising placement effectiveness and affect the return on investment (ROI) of advertisers and user experience. Specifically, the paper points out that due to the rapid change of data distribution and the inherent deviation of the model, there is often a mismatch between the probabilities predicted by the model and the real probabilities. In addition, for some characteristic field values, the observed sample size may be very limited, which makes it difficult to perform confidence calibration based on these fields and may lead to deviation amplification and online interference. To solve these problems, the paper proposes a new method - ConfCalib, that is, a multi - field model calibration method based on confidence. This method can adaptively adjust the calibration intensity according to sample statistical information and use multiple fields to jointly calibrate the model output to reduce the impact of data sparsity in a single field. Through this method, the paper aims to improve advertising performance and reduce prediction deviation. Specifically, the main contributions of ConfCalib include: 1. Proposing a model calibration method based on confidence, which can adaptively calibrate the predictions of the recommendation model while considering the data sparsity of fields. 2. Introducing a simple and effective multi - field joint calibration method, which improves the calibration performance and robustness to data sparsity. 3. Verifying the effectiveness of this method through experiments on different offline data sets and online A/B tests on the advertising platform. The method in the paper shows stronger robustness in dealing with data sparsity, especially in the case of a small sample size or zero positive samples, and can maintain a good calibration effect. This makes ConfCalib have higher practical value in practical applications.

Confidence-Aware Multi-Field Model Calibration

Field-aware Calibration: A Simple and Empirically Strong Method for Reliable Probabilistic Predictions

Deep Ensemble Shape Calibration: Multi-Field Post-hoc Calibration in Online Advertising

Confidence Calibration for Intent Detection Via Hyperspherical Space and Rebalanced Accuracy-Uncertainty Loss

Two Sides of Miscalibration: Identifying Over and Under-Confidence Prediction for Network Calibration

From Uncertainty to Precision: Enhancing Binary Classifier Performance through Calibration

The Calibration Gap between Model and Human Confidence in Large Language Models

A Self-boosted Framework for Calibrated Ranking

Confidence Calibration and Rationalization for LLMs via Multi-Agent Deliberation

Human-Aligned Calibration for AI-Assisted Decision Making

Multi-Head Multi-Loss Model Calibration

Optimizing Calibration by Gaining Aware of Prediction Correctness

Confidence Calibration for Domain Generalization under Covariate Shift

On Calibrated Predictions for Auction Selection Mechanisms

Uncertainty Calibration for Counterfactual Propensity Estimation in Recommendation

Calibrating Multimodal Learning

Re-Examining Calibration: The Case of Question Answering

Operational Calibration: Debugging Confidence Errors for DNNs in the Field

Confidence Calibration of Classifiers with Many Classes

Probabilistic Scores of Classifiers, Calibration is not Enough

Calibration-then-Calculation: A Variance Reduced Metric Framework in Deep Click-Through Rate Prediction Models