Abstract:Although the responses of dopamine neurons in the primate midbrain are well characterized as carrying a temporal difference (TD) error signal for reward prediction, existing theories do not offer a credible account of how the brain keeps track of past sensory events that may be relevant to predicting future reward. Empirically, these shortcomings of previous theories are particularly evident in their account of experiments in which animals were exposed to variation in the timing of events. The original theories mispredicted the results of such experiments due to their use of a representational device called a tapped delay line. Here we propose that a richer understanding of history representation and a better account of these experiments can be given by considering TD algorithms for a formal setting that incorporates two features not originally considered in theories of the dopaminergic response: partial observability (a distinction between the animal's sensory experience and the true underlying state of the world) and semi-Markov dynamics (an explicit account of variation in the intervals between events). The new theory situates the dopaminergic system in a richer functional and anatomical context, since it assumes (in accord with recent computational theories of cortex) that problems of partial observability and stimulus history are solved in sensory cortex using statistical modeling and inference and that the TD system predicts reward using the results of this inference rather than raw sensory data. It also accounts for a range of experimental data, including the experiments involving programmed temporal variability and other previously unmodeled dopaminergic response phenomena, which we suggest are related to subjective noise in animals' interval timing. Finally, it offers new experimental predictions and a rich theoretical framework for designing future experiments.

Dopamine transients follow a striatal gradient of reward time horizons

Dopamine transients encode reward prediction errors independent of learning rates

Subsecond dopamine fluctuations in human striatum encode superposed error signals about actual and counterfactual reward

Dopamine transients delivered in learning contexts do not act as model-free prediction errors

Causal evidence supporting the proposal that dopamine transients function as temporal difference prediction errors

Dopamine transients do not act as model-free prediction errors during associative learning

Intrinsic dopamine and acetylcholine dynamics in the striatum of mice

Does phasic dopamine release cause policy updates?

Intrinsic Reward-Like Dopamine and Acetylcholine Dynamics in Striatum

Behavioral encoding across timescales by region-specific dopamine dynamics

Representation and Timing in Theories of the Dopamine System

Slowly Evolving Dopaminergic Activity Modulates the Moment-to-moment Probability of Reward-Related Self-Timed Movements

Dopamine neurons drive spatiotemporally heterogeneous striatal dopamine signals during learning

Subsecond fluctuations in extracellular dopamine encode reward and punishment prediction errors in humans

Expected reward value and reward prediction errors reinforce but also interfere with human time perception

Dopamine reward prediction error signal codes the temporal evaluation of a perceptual decision report

Dopaminergic prediction errors in the ventral tegmental area reflect a multithreaded predictive model

Mesolimbic dopamine encodes reward prediction errors independent of learning rates

Dopamine in the dorsal bed nucleus of stria terminalis signals Pavlovian sign-tracking and reward violations

Computational modelling identifies key determinants of subregion-specific dopamine dynamics in the striatum