Abstract:Evidence accumulations models (EAMs) have become the dominant modeling framework within rapid decision-making, using choice response time distributions to make inferences about the underlying decision process. These models are often applied to empirical data as “measurement tools”, with different theoretical accounts being contrasted within the framework of the model. Some method is then needed to decide between these competing theoretical accounts, as only assessing the models on their ability to fit trends in the empirical data ignores model flexibility, and therefore, creates a bias towards more flexible models. However, there is no objectively optimal method to select between models, with methods varying in both their computational tractability and theoretical basis. I provide a systematic comparison between nine different model selection methods using a popular EAM—the linear ballistic accumulator (LBA; Brown & Heathcote, Cognitive Psychology 57(3), 153–178 2008)—in a large-scale simulation study and the empirical data of Dutilh et al. (Psychonomic Bulletin and Review, 1–19 2018). I find that the “predictive accuracy” class of methods (i.e., the Akaike Information Criterion [AIC], the Deviance Information Criterion [DIC], and the Widely Applicable Information Criterion [WAIC]) make different inferences to the “Bayes factor” class of methods (i.e., the Bayesian Information Criterion [BIC], and Bayes factors) in many, but not all, instances, and that the simpler methods (i.e., AIC and BIC) make inferences that are highly consistent with their more complex counterparts. These findings suggest that researchers should be able to use simpler “parameter counting” methods when applying the LBA and be confident in their inferences, but that researchers need to carefully consider and justify the general class of model selection method that they use, as different classes of methods often result in different inferences.

A brief guide to model selection, multimodel inference and model averaging in behavioural ecology using Akaike’s information criterion

AIC model selection and multimodel inference in behavioral ecology: some background, observations, and comparisons

Uninformative Parameters and Model Selection Using Akaike's Information Criterion

Introducing AIC model averaging in ecological niche modeling: a single-algorithm multi-model strategy to account for uncertainty in suitability predictions

Model selection and psychological theory: A discussion of the differences between the Akaike information criterion (AIC) and the Bayesian information criterion (BIC).

A practical guide to selecting models for exploration, inference, and prediction in ecology

Assessing the Significance of Model Selection in Ecology

AIC under the framework of least squares estimation

A brief introduction to mixed effects modelling and multi-model inference in ecology

Bayesian Model Selection for a Class of Spatially-Explicit Capture Recapture Models

Comparision of AIC and BIC in the Selection of Stock-Recruitment Relationships

The Focused Information Criterion for Stochastic Model Selection Problems Using $M$-Estimators

Extending AIC to best subset regression

A simple application of FIC to model selection

Assessing the practical differences between model selection methods in inferences about choice response time tasks

Monte Carlo algorithms for model assessment via conflicting summaries

A biologist's guide to model selection and causal inference

Posterior Averaging Information Criterion

Factor Analysis and AIC

Cross validation for model selection: a primer with examples from ecology

Multimodel Inference Based on Smoothed Information Criteria