Abstract:Abstract In everyday life, our behavior varies on a continuum from automatic and habitual to deliberate and goal-directed. Recent evidence suggests that habit formation and relearning of habits operate in a context-dependent manner: Habit formation is promoted when actions are performed in a specific context, while breaking off habits is facilitated after a context change. It is an open question how one can computationally model the brain’s balancing between context-specific habits and goal-directed actions. Here, we propose a hierarchical Bayesian approach for control of a partially observable Markov decision process that enables conjoint learning of habits and reward structure in a context-specific manner. In this model, habit learning corresponds to an updating of priors over policies and interacts with the learning of the outcome contingencies. Importantly, the model is solely built on probabilistic inference, which effectively provides a simple explanation of how the brain may balance contributions of habitual and goal-directed control. We illustrated the resulting behavior using agent-based simulated experiments, where we replicated several findings of devaluation, extinction, and renewal experiments, as well as the so-called two-step task which is typically used with human participants. In addition, we show how a single parameter, the habitual tendency, can explain individual differences in habit learning and the balancing between habitual and goal-directed control. Finally, we discuss the link of the proposed model to other habit learning models and implications for understanding specific phenomena in substance use disorder.

Tracking subjects' strategies in behavioural choice experiments at trial resolution

Tracking subjects’ strategies in behavioural choice experiments at trial resolution

Extracting the dynamics of behavior in sensory decision-making experiments

Simple Plans or Sophisticated Habits? State, Transition and Learning Interactions in the Two-Step Task

Competing neural representations of choice shape evidence accumulation in humans

Learning to Choose: Behavioral Dynamics Underlying the Initial Acquisition of Decision Making

Learning to Choose: Behavioral Dynamics Underlying the Initial Acquisition of Decision-Making

Learning and choosing in an uncertain world: An investigation of the explore-exploit dilemma in static and dynamic environments

The expression of decision and learning variables in movement patterns related to decision actions

Tracking the evolution of a single choice

Striatal dopamine reflects individual long-term learning trajectories

Instance-based learning: Integrating sampling and repeated decisions from experience.

One-shot learning and behavioral eligibility traces in sequential decision making

Signals in Human Striatum Are Appropriate for Policy Update Rather Than Value Prediction

Time-evolving psychological processes over repeated decisions

From Learning to Choosing: How Decision-Making Evolves with Experience in Rats

Adapting non-invasive human recordings along multiple task-axes shows unfolding of spontaneous and over-trained choice

Humans rationally balance detailed and temporally abstract world models

Trial-history biases in evidence accumulation can give rise to apparent lapses in decision-making

Balancing control: a Bayesian interpretation of habitual and goal-directed behavior

How cortico-basal ganglia-thalamic subnetworks can shift decision policies to maximize reward rate