Abstract:Abstract Sequential sampling models such as the drift diffusion model have a long tradition in research on perceptual decision-making, but mounting evidence suggests that these models can account for response time distributions that arise during reinforcement learning and value-based decision-making. Building on this previous work, we implemented the drift diffusion model as the choice rule in inter-temporal choice ( temporal discounting ) and risky choice ( probability discounting ) using a hierarchical Bayesian estimation scheme. We validated our approach in data from nine patients with focal lesions to the ventromedial prefrontal cortex / medial orbitofrontal cortex (vmPFC/mOFC) and nineteen age- and education-matched controls. Choice model parameters estimated via standard softmax action selection were reliably reproduced using the drift diffusion model as the choice rule, both for temporal discounting and risky choice. Model comparison revealed that, for both tasks, the data were best accounted for by a variant of the drift diffusion model including a non-linear mapping from value-differences to trial-wise drift rates. Posterior predictive checks of the winning models revealed a reasonably good fit to individual participants reaction time distributions. We then applied this modeling framework and 1) reproduced our previous results regarding temporal discounting in vmPFC/mOFC patients and 2) showed in a previously unpublished data set on risky choice that vmPFC/mOFC patients exhibit increased risk-taking relative to controls. Analyses of diffusion model parameters revealed that vmPFC/mOFC damage abolished neither value sensitivity nor asymptote of the drift rate. Rather, it substantially increased non-decision times and reduced response caution during risky choice. Our results highlight that novel insights can be gained from applying sequential sampling models in studies of inter-temporal and risky decision-making in cognitive neuroscience.

A reinforcement learning diffusion decision model for value-based decisions

The drift diffusion model as the choice rule in reinforcement learning

Learning to Choose: Behavioral Dynamics Underlying the Initial Acquisition of Decision-Making

The drift diffusion model as the choice rule in inter-temporal and risky choice: a case study in medial orbitofrontal cortex lesion patients and controls

Learning to Choose: Behavioral Dynamics Underlying the Initial Acquisition of Decision Making

Decision-Focused Model-based Reinforcement Learning for Reward Transfer

Diffusion Spectral Representation for Reinforcement Learning

Extracting Reward Functions from Diffusion Models

A Behavioral Characterization of the Drift Diffusion Model and Its Multialternative Extension for Choice Under Time Pressure

Training Diffusion Models with Reinforcement Learning

Diffusion World Model: Future Modeling Beyond Step-by-Step Rollout for Offline Reinforcement Learning

Reward Shaping via Diffusion Process in Reinforcement Learning

A model of discrete choice based on reinforcement learning under short-term memory

The Tweedledum and Tweedledee of dynamic decisions: Discriminating between diffusion decision and accumulator models

HMM for Discovering Decision-Making Dynamics Using Reinforcement Learning Experiments

Reinforcement Learning in Non-Markov Market-Making

Diffusion Models for Reinforcement Learning: A Survey

Embodied sequential sampling models and dynamic neural fields for decision-making: Why hesitate between two when a continuum is the answer

CM-DQN: A Value-Based Deep Reinforcement Learning Model to Simulate Confirmation Bias

The role of reinforcement learning in shaping the decision policy in methamphetamine use disorders