Abstract:Although the responses of dopamine neurons in the primate midbrain are well characterized as carrying a temporal difference (TD) error signal for reward prediction, existing theories do not offer a credible account of how the brain keeps track of past sensory events that may be relevant to predicting future reward. Empirically, these shortcomings of previous theories are particularly evident in their account of experiments in which animals were exposed to variation in the timing of events. The original theories mispredicted the results of such experiments due to their use of a representational device called a tapped delay line. Here we propose that a richer understanding of history representation and a better account of these experiments can be given by considering TD algorithms for a formal setting that incorporates two features not originally considered in theories of the dopaminergic response: partial observability (a distinction between the animal's sensory experience and the true underlying state of the world) and semi-Markov dynamics (an explicit account of variation in the intervals between events). The new theory situates the dopaminergic system in a richer functional and anatomical context, since it assumes (in accord with recent computational theories of cortex) that problems of partial observability and stimulus history are solved in sensory cortex using statistical modeling and inference and that the TD system predicts reward using the results of this inference rather than raw sensory data. It also accounts for a range of experimental data, including the experiments involving programmed temporal variability and other previously unmodeled dopaminergic response phenomena, which we suggest are related to subjective noise in animals' interval timing. Finally, it offers new experimental predictions and a rich theoretical framework for designing future experiments.

Dopamine neurons code subjective sensory experience and uncertainty of perceptual decisions

Dopamine reward prediction error signal codes the temporal evaluation of a perceptual decision report

Dopamine firing plays a double role in coding reward prediction errors and signaling motivation in a working memory task

Dopamine, Inference, and Uncertainty

The Dopaminergic Midbrain Encodes the Expected Certainty about Desired Outcomes

Dopamine neurons encode trial-by-trial subjective reward value in an auction-like task

Dopamine prediction error responses integrate subjective value from different reward dimensions

Dopamine neuron ensembles signal the content of sensory prediction errors

Human Substantia Nigra Neurons Encode Reward Expectations

Believing in dopamine

Dopamine neurons report an error in the temporal prediction of reward during learning

Dopamine neurons learn to encode the long-term value of multiple future rewards

Subsecond dopamine fluctuations in human striatum encode superposed error signals about actual and counterfactual reward

Dopamine firing plays a dual role in coding reward prediction errors and signaling motivation in a working memory task

Representation and Timing in Theories of the Dopamine System

Subsecond fluctuations in extracellular dopamine encode reward and punishment prediction errors in humans

Dynamic coding and sequential integration of multiple reward attributes by primate amygdala neurons

Distinct dynamics and intrinsic properties in ventral tegmental area populations mediate reward association and motivation

Dopamine reward prediction error coding

Dopamine, Prediction Error and Beyond