Abstract:A large literature has accumulated suggesting that human and animal decision making is driven by at least two systems, and that important functions of these systems can be captured by reinforcement learning algorithms. The "model-free" system caches and uses stimulus–value or stimulus–response associations, and the "model-based" system implements more flexible planning using a model of the world. However, it is not clear how the two systems interact during deliberation and how a single decision emerges from this process, especially when they disagree. Most previous work has assumed that while the systems operate in parallel, they do so independently, and they combine linearly to influence decisions. Using an integrated reinforcement learning/drift-diffusion model, we tested the hypothesis that the two systems interact in a non-linear fashion similar to other situations with cognitive conflict. We differentiated two forms of conflict: action conflict , a binary state representing whether the systems disagreed on the best action, and value conflict , a continuous measure of the extent to which the two systems disagreed on the difference in value between the available options. We found that decisions with greater value conflict were characterized by reduced model-based control and increased caution both with and without action conflict. Action conflict itself (the binary state) acted in the opposite direction, although its effects were less prominent. We also found that between-system conflict was highly correlated with within-system conflict, and although it is less clear a priori why the latter might influence the strength of each system above its standard linear contribution, we could not rule it out. Our work highlights the importance of non-linear conflict effects, and provides new constraints for more detailed process models of decision making. It also presents new avenues to explore with relation to disorders of compulsivity, where an imbalance between systems has been implicated. A number of studies have framed goal-directed and habitual decision making from the perspective of different reinforcement learning algorithms ("model-based" and "model-free"), and further suggested that they are supported by separate though potentially overlapping systems. However, there has been little work to understand how the different systems work together. By design, they will sometimes disagree on the identity of the best action, and even when they agree, they will assign different values to the actions. Despite this, the end result is a single behavioral output. The issue of how the two systems interact and compete draws parallels to the existing literature on cognitive control, where a central question has been how more 'costly' cognitive resources should be deployed in the presence of decision conflict (here, the goal-directed system is more computationally 'expensive'). Across four datasets, we found that the influence of the goal-directed system was reduced as a function of conflict between systems, and in addition, responses overall were more cautious. Our results provide new constraints for process models of decision making, and suggest new research directions for questions related to psychopathology and disorders of compulsivity in particular, where an imbalance between the two systems has previously been implicated.

LATENT SUBSTRATES OF COMPULSIVITY AND BEHAVIOURAL FLEXIBILITY

Disorders of Compulsivity: a Common Bias Towards Learning Habits.

Conflict and competition between model-based and model-free control

Disorders of compulsivity: Deficits in arbitrating learning strategies

Computational Modelling of Reinforcement Learning and Functional Neuroimaging of Probabilistic Reversal for Dissociating Compulsive Behaviours in Gambling and Cocaine Use Disorders

Fronto-striatal Organization: Defining Functional and Microstructural Substrates of Behavioural Flexibility

Contingency-based flexibility mechanisms through a reinforcement learning model in adults with Attention-Deficit/Hyperactivity Disorder and Obsessive-Compulsive Disorder

Computational modeling of reinforcement learning and functional neuroimaging of probabilistic reversal dissociates compulsive behaviors in Gambling and Cocaine Use Disorders

Learning and Cognitive Flexibility: Frontostriatal Function and Monoaminergic Modulation

Interactions Among Working Memory, Reinforcement Learning, and Effort in Value-Based Choice: A New Paradigm and Selective Deficits in Schizophrenia

Neurocomputational model of compulsivity: Deviating from an uncertain goal-directed system

Balancing control: a Bayesian interpretation of habitual and goal-directed behavior

Information Processing in Decision-Making Systems

Model of a striatal circuit exploring biological mechanisms underlying decision-making during normal and disordered states

Computational modeling of decision-making in substance abusers: testing Bechara's hypotheses

On the effects of impulsivity and compulsivity on neural correlates of model-based performance

Computational modelling reveals contrasting effects on reinforcement learning and cognitive flexibility in stimulant use disorder and obsessive-compulsive disorder: remediating effects of dopaminergic D2/3 receptor agents

From compulsivity to compulsion: the neural basis of compulsive disorders

Goal-directed learning and obsessive-compulsive disorder.

Computational modelling reveals contrasting effects on reinforcement learning and cognitive flexibility in stimulant use disorder and obsessive compulsive disorder: remediating …