Abstract:Learning to make adaptive decisions involves making choices, assessing their consequence, and leveraging this assessment to attain higher rewarding states. Despite vast literature on value-based decision-making, relatively little is known about the cognitive processes underlying decisions in highly uncertain contexts. Real world decisions are rarely accompanied by immediate feedback, explicit rewards, or complete knowledge of the environment. Being able to make informed decisions in such contexts requires significant knowledge about the environment, which can only be gained via exploration. Here we aim at understanding and formalizing the brain mechanisms underlying these processes. To this end, we first designed and performed an experimental task. Human participants had to learn to maximize reward while making sequences of decisions with only basic knowledge of the environment, and in the absence of explicit performance cues. Participants had to rely on their own internal assessment of performance to reveal a covert relationship between their choices and their subsequent consequences to find a strategy leading to the highest cumulative reward. Our results show that the participants' reaction times were longer whenever the decision involved a future consequence, suggesting greater introspection whenever a delayed value had to be considered. The learning time varied significantly across participants. Second, we formalized the neurocognitive processes underlying decision-making within this task, combining mean-field representations of competing neural populations with a reinforcement learning mechanism. This model provided a plausible characterization of the brain dynamics underlying these processes, and reproduced each aspect of the participants' behavior, from their reaction times and choices to their learning rates. In summary, both the experimental results and the model provide a principled explanation to how delayed value may be computed and incorporated into the neural dynamics of decision-making, and to how learning occurs in these uncertain scenarios.

Continuous decision to wait for a future reward is guided by fronto-hippocampal anticipatory dynamics

Intertemporal Decision-Making Involves Prefrontal Control Mechanisms Associated with Working Memory

Executive control by fronto-parietal activity explains counterintuitive decision behavior in complex value-based decision-making

Neural substrates of the interaction between effort-expenditure reward decision-making and outcome anticipation

A role for the midbrain reticular formation in delay-based decision making

Policy adjustment in a dynamic economic game

Cognitive mechanisms of learning in sequential decision-making under uncertainty: an experimental and theoretical approach

Overlapping Neural Systems Represent Cognitive Effort and Reward Anticipation

The medial prefrontal cortex leaves the hippocampus when it prepares for the future

Reward-based option competition in human dorsal stream and transition from stochastic exploration to exploitation in continuous space

The neural substrates of probabilistic and intertemporal decision making

Impulsive Choices Emerge When the Anterior Cingulate Cortex Fails to Encode Deliberative Strategies

Anterior cingulate cortex provides the neural substrates for feedback-driven iteration of decision and value representation

Decisions in Motion: Decision Dynamics during Intertemporal Choice reflect Subjective Evaluation of Delayed Rewards

Balancing safety and efficiency in human decision making

Differential phase-amplitude coupling in nucleus accumbens and orbitofrontal cortex reflects decision-making during a delay discounting task

The Preference for Future Outcomes Correlates with the Temporal Variability of Functional Connectivity among Brain Regions

Activity of Prefrontal Neurons Predict Future Choices during Gambling

The Neural Predictors of Choice Preference in Intertemporal Choice

Neurophysiological insights into sequential decision-making: exploring the secretary problem through ERPs and TBR dynamics

A common stay-on-goal mechanism in anterior cingulate cortex for information and effort choices