Abstract:Animals can adapt their preferences for different types of reward according to physiological state, such as hunger or thirst. To explain this ability, we employ a simple multi-objective reinforcement learning model that learns multiple values according to different reward dimensions such as food or water. We show that by weighting these learned values according to the current needs, behaviour may be flexibly adapted to present preferences. This model predicts that individual dopamine neurons should encode the errors associated with some reward dimensions more than with others. To provide a preliminary test of this prediction, we reanalysed a small dataset obtained from a single primate in an experiment which to our knowledge is the only published study where the responses of dopamine neurons to stimuli predicting distinct types of rewards were recorded. We observed that in addition to subjective economic value, dopamine neurons encode a gradient of reward dimensions; some neurons respond most to stimuli predicting food rewards while the others respond more to stimuli predicting fluids. We also proposed a possible implementation of the model in the basal ganglia network, and demonstrated how the striatal system can learn values in multiple dimensions, even when dopamine neurons encode mixtures of prediction error from different dimensions. Additionally, the model reproduces the instant generalisation to new physiological states seen in dopamine responses and in behaviour. Our results demonstrate how a simple neural circuit can flexibly guide behaviour according to animals' needs. Animals and humans can search for different resources depending on their needs. For example, when you are thirsty at work, you may go to a common room where hopefully coffee or water is available, while if you are hungry, you would rather go to a canteen. Such ability to seek different resources based on a physiological state is so fundamental to survival, that is present also in simple animals. This paper proposes how this ability could arise from a simple neural circuit that can be mapped on evolutionary older parts of the vertebrate brain, called the basal ganglia. The model suggests that this circuit learns the availability of different reward types, and then combines them according to the physiological state to control behaviour.

Adaptation of reward sensitivity in orbitofrontal neurons

Single-cell and population coding of expected reward probability in the orbitofrontal cortex of the rat

Cracking and Packing Information about the Features of Expected Rewards in the Orbitofrontal Cortex

Reward Bases: A simple mechanism for adaptive acquisition of multiple reward types

The Neural Dynamics of Reward Value and Risk Coding in the Human Orbitofrontal Cortex.

Synaptic and Spiking Dynamics Underlying Reward Reversal in the Orbitofrontal Cortex.

Neurons in rat orbitofrontal cortex and medial prefrontal cortex exhibit distinct responses in reward and strategy-update in a risk-based decision-making task

Neural Representations of Post-Decision Accuracy and Reward Expectation in the Caudate Nucleus and Frontal Eye Field

Stimulus-dependent adjustment of reward prediction error in the midbrain

Adaptive coding of reward in schizophrenia, its change over time and relation to apathy

Efficient value synthesis in the orbitofrontal cortex explains how loss aversion adapts to the ranges of gain and loss prospects

Orbitofrontal Cortex Mediates Sustained Basolateral Amygdala Encoding of Cued Reward-Seeking States

Reward Prediction Based on Stimulus Categorization in Primate Lateral Prefrontal Cortex

Efficient coding explains neural response homeostasis and stimulus-specific adaptation

Predictive Coding of Reward in the Hippocampus

Human Orbitofrontal Cortex Signals Decision Outcomes to Sensory Cortex During Behavioral Adaptations

Dopaminergic responses to identity prediction errors depend differently on the orbitofrontal cortex and hippocampus

Neural dynamics in the orbitofrontal cortex reveal cognitive strategies

Adaptation optimizes sensory encoding of future stimuli

Dopamine, Inference, and Uncertainty

How cortico-basal ganglia-thalamic subnetworks can shift decision policies to maximize reward rate