Abstract:Phasic activity of midbrain dopamine neurons is currently thought to encapsulate the prediction-error signal described in Sutton and Barto's (1981) model-free reinforcement learning algorithm. This phasic signal is thought to contain information about the quantitative value of reward, which transfers to the reward-predictive cue after learning. This is argued to endow the reward-predictive cue with the value inherent in the reward, motivating behavior toward cues signaling the presence of reward. Yet theoretical and empirical research has implicated prediction-error signaling in learning that extends far beyond a transfer of quantitative value to a reward-predictive cue. Here, we review the research which demonstrates the complexity of how dopaminergic prediction errors facilitate learning. After briefly discussing the literature demonstrating that phasic dopaminergic signals can act in the manner described by Sutton and Barto (1981), we consider how these signals may also influence attentional processing across multiple attentional systems in distinct brain circuits. Then, we discuss how prediction errors encode and promote the development of context-specific associations between cues and rewards. Finally, we consider recent evidence that shows dopaminergic activity contains information about causal relationships between cues and rewards that reflect information garnered from rich associative models of the world that can be adapted in the absence of direct experience. In discussing this research we hope to support the expansion of how dopaminergic prediction errors are thought to contribute to the learning process beyond the traditional concept of transferring quantitative value.

Nucleus accumbens dopamine release reflects Bayesian inference during instrumental learning

Believing in dopamine

Dopamine, Inference, and Uncertainty

Mesolimbic dopamine encodes reward prediction errors independent of learning rates

The Dopaminergic Midbrain Encodes the Expected Certainty about Desired Outcomes

Dynamic shaping of dopamine signals during probabilistic Pavlovian conditioning

Dopamine Release in the Nucleus Accumbens Core Encodes the General Excitatory Components of Learning

Dopamine release in the nucleus accumbens core signals perceived saliency

Dopamine transients encode reward prediction errors independent of learning rates

The Dopamine Prediction Error: Contributions to Associative Models of Reward Learning

Midbrain dopamine neurons compute inferred and cached value prediction errors in a common framework

Model-based predictions for dopamine

Subsecond fluctuations in extracellular dopamine encode reward and punishment prediction errors in humans

Model-based reward prediction in the primate prefrontal cortex

Dopaminergic prediction errors in the ventral tegmental area reflect a multithreaded predictive model

Dopamine Release Plateau and Outcome Signals in Dorsal Striatum Contrast with Classic Reinforcement Learning Formulations

A causal link between prediction errors, dopamine neurons and learning

Dopamine reveals adaptive learning of actions representation

A Causal Role for the Pedunculopontine Nucleus in Human Instrumental Learning