Abstract:Perceptual confidence results from a metacognitive process which evaluates how likely our percepts are to be correct. Many competing models of perceptual metacognition enjoy strong empirical support. Arbitrating these models traditionally proceeds via researchers conducting experiments and then fitting several models to the data collected. However, such a process often includes conditions or paradigms that may not best arbitrate competing models: Many models make similar predictions under typical experimental conditions. Consequently, many experiments are needed, collectively (sub-optimally) sampling the space of conditions to compare models. Here, instead, we introduce a variant of optimal experimental design which we call a computational-rationality approach to generative models of cognition, using perceptual metacognition as a case study. Instead of designing experiments and post-hoc specifying models, we began with comprehensive model comparison among four competing generative models for perceptual metacognition, drawn from literature. By simulating a simple experiment under each model, we identified conditions where these models made maximally diverging predictions for confidence. We then presented these conditions to human observers, and compared the models' capacity to predict choices and confidence. Results revealed two surprising findings: (1) two models previously reported to differently predict confidence to different degrees, with one predicting better than the other, appeared to predict confidence in a direction opposite to previous findings; and (2) two other models previously reported to equivalently predict confidence showed stark differences in the conditions tested here. Although preliminary with regards to which model is actually 'correct' for perceptual metacognition, our findings reveal the promise of this computational-rationality approach to maximizing experimental utility in model arbitration while minimizing the number of experiments necessary to reveal the winning model, both for perceptual metacognition and in other domains.

The Confidence-Noise Confidence-Boost (CNCB) model of confidence rating data

Modeling perceptual confidence and the confidence forced-choice paradigm.

Toward 'Computational-Rationality' Approaches to Arbitrating Models of Cognition: A Case Study Using Perceptual Metacognition

Modeling confidence judgments, response times, and multiple choices in decision making: Recognition memory and motion discrimination.

MACEst: The reliable and trustworthy Model Agnostic Confidence Estimator

The Sense of Confidence during Probabilistic Learning: A Normative Account

On the connection between Noise-Contrastive Estimation and Contrastive Divergence

On the assumptions behind metacognitive measurements: Implications for theory and practice

Two Sides of Miscalibration: Identifying Over and Under-Confidence Prediction for Network Calibration

Beyond Calibration: Assessing the Probabilistic Fit of Neural Regressors via Conditional Congruence

Confidence-Aware Calibration and Scoring Functions for Curriculum Learning

Dynamic noise estimation: A generalized method for modeling noise fluctuations in decision-making

From Conformal Predictions to Confidence Regions

Neurophysiology of perceived confidence

Noise-Adaptive Confidence Sets for Linear Bandits and Application to Bayesian Optimization

Confidence control for efficient behaviour in dynamic environments

Confidence Is the Bridge between Multi-stage Decisions

Cycles of Thought: Measuring LLM Confidence through Stable Explanations

Confidence is detection-like in high-dimensional spaces

Neuroscientific User Models: The Source of Uncertain User Feedback and Potentials for Improving Recommendation and Personalisation