Abstract:Perceptual confidence results from a metacognitive process which evaluates how likely our percepts are to be correct. Many competing models of perceptual metacognition enjoy strong empirical support. Arbitrating these models traditionally proceeds via researchers conducting experiments and then fitting several models to the data collected. However, such a process often includes conditions or paradigms that may not best arbitrate competing models: Many models make similar predictions under typical experimental conditions. Consequently, many experiments are needed, collectively (sub-optimally) sampling the space of conditions to compare models. Here, instead, we introduce a variant of optimal experimental design which we call a computational-rationality approach to generative models of cognition, using perceptual metacognition as a case study. Instead of designing experiments and post-hoc specifying models, we began with comprehensive model comparison among four competing generative models for perceptual metacognition, drawn from literature. By simulating a simple experiment under each model, we identified conditions where these models made maximally diverging predictions for confidence. We then presented these conditions to human observers, and compared the models' capacity to predict choices and confidence. Results revealed two surprising findings: (1) two models previously reported to differently predict confidence to different degrees, with one predicting better than the other, appeared to predict confidence in a direction opposite to previous findings; and (2) two other models previously reported to equivalently predict confidence showed stark differences in the conditions tested here. Although preliminary with regards to which model is actually 'correct' for perceptual metacognition, our findings reveal the promise of this computational-rationality approach to maximizing experimental utility in model arbitration while minimizing the number of experiments necessary to reveal the winning model, both for perceptual metacognition and in other domains.

The Importance of Falsification in Computational Cognitive Modeling

The Experiment is just as Important as the Likelihood in Understanding the Prior: a Cautionary Note on Robust Cognitive Modeling

Model Comparison in Cognitive Modeling

Formalizing Falsification for Theories of Consciousness Across Computational Hierarchies

Falsification and consciousness

The experiment is just as important as the likelihood in understanding the prior: A cautionary note on robust cognitive modelling

Validation and Comparison of Non-Stationary Cognitive Models: A Diffusion Model Application

Cognitive Models as Simulators: The Case of Moral Decision-Making

We Ran 9 Billion Regressions: Eliminating False Positives through Computational Model Robustness

Toward 'Computational-Rationality' Approaches to Arbitrating Models of Cognition: A Case Study Using Perceptual Metacognition

No Substitute for Functionalism -- A Reply to 'Falsification & Consciousness'

Robust Modeling in Cognitive Science

Online simulator-based experimental design for cognitive model selection

From evolutionary ecosystem simulations to computational models of human behavior

Falsification of Autonomous Systems in Rich Environments

Exposing Assumptions in AI Benchmarks through Cognitive Modelling

When a good fit can be bad

Pitfalls and potentials in simulation studies: Questionable research practices in comparative simulation studies allow for spurious claims of superiority of any method

Optimizing Falsification for Learning-Based Control Systems: A Multi-Fidelity Bayesian Approach

A Workflow for Building Computationally Rational Models of Human Behavior

Parameters, Predictions, and Evidence in Computational Modeling: A Statistical View Informed by ACT-R