Abstract:Perceptual confidence results from a metacognitive process which evaluates how likely our percepts are to be correct. Many competing models of perceptual metacognition enjoy strong empirical support. Arbitrating these models traditionally proceeds via researchers conducting experiments and then fitting several models to the data collected. However, such a process often includes conditions or paradigms that may not best arbitrate competing models: Many models make similar predictions under typical experimental conditions. Consequently, many experiments are needed, collectively (sub-optimally) sampling the space of conditions to compare models. Here, instead, we introduce a variant of optimal experimental design which we call a computational-rationality approach to generative models of cognition, using perceptual metacognition as a case study. Instead of designing experiments and post-hoc specifying models, we began with comprehensive model comparison among four competing generative models for perceptual metacognition, drawn from literature. By simulating a simple experiment under each model, we identified conditions where these models made maximally diverging predictions for confidence. We then presented these conditions to human observers, and compared the models' capacity to predict choices and confidence. Results revealed two surprising findings: (1) two models previously reported to differently predict confidence to different degrees, with one predicting better than the other, appeared to predict confidence in a direction opposite to previous findings; and (2) two other models previously reported to equivalently predict confidence showed stark differences in the conditions tested here. Although preliminary with regards to which model is actually 'correct' for perceptual metacognition, our findings reveal the promise of this computational-rationality approach to maximizing experimental utility in model arbitration while minimizing the number of experiments necessary to reveal the winning model, both for perceptual metacognition and in other domains.

When a good fit can be bad

Alternative Ways of Assessing Model Fit

A Goodness-of-Fit Test for Statistical Models

Model Selection Techniques -- An Overview

Revisiting the importance of model fitting for model-based fMRI: It does matter in computational psychiatry

Testing for Overfitting

Parsimony in Model Selection: Tools for Assessing Fit Propensity

The Accuracy of Bayesian Model Fit Indices in Selecting Among Multidimensional Item Response Theory Models

Multi-model mimicry for model selection according to generalised goodness-of-fit criteria

Assessing the Significance of Model Selection in Ecology

Reply to "Comment on 'Experimentally adjudicating between different causal accounts of Bell-inequality violations via statistical model selection'"

Prediction-Oriented Model Selection In Partial Least Squares Path Modeling

The Importance of Falsification in Computational Cognitive Modeling

Selecting a classification performance measure: matching the measure to the problem

Performance of Model Fit and Selection Indices for Bayesian Structural Equation Modeling with Missing Data

Toward 'Computational-Rationality' Approaches to Arbitrating Models of Cognition: A Case Study Using Perceptual Metacognition

Model Fit and Item Factor Analysis: Overfactoring, Underfactoring, and a Program to Guide Interpretation

Machine learning models and over-fitting considerations

Tests of goodness of fit to multiple data sets

Model-Selection-Based Approaches to Identifying the Optimal Number of Factors in Multilevel Exploratory Factor Analysis

The Poor Fit of Model Fit for Selecting Number of Factors in Exploratory Factor Analysis for Scale Evaluation