Abstract:The structural credit assignment problem arises when the causal structure between actions and subsequent outcomes is hidden from direct observation. To solve this problem and enable goal-directed behavior, an agent has to infer structure and form a representation thereof. In the scope of this study, we investigate a possible solution in the human brain. We recorded behavioral and electrophysiological data from human participants in a novel variant of the bandit task, where multiple actions lead to multiple outcomes. Crucially, the mapping between actions and outcomes was hidden and not instructed to the participants. Human choice behavior revealed clear hallmarks of credit assignment and learning. Moreover, a computational model which formalizes action selection as the competition between multiple representations of the hidden structure was fit to account for participants data. Starting in a state of uncertainty about the correct representation, the central mechanism of this model is the arbitration of action control towards the representation which minimizes surprise about outcomes. Crucially, single-trial latent-variable analysis reveals that the neural patterns clearly support central quantitative predictions of this surprise minimization model. The results suggest that neural activity is not only related to reinforcement learning under correct as well as incorrect task representations but also reflects central mechanisms of credit assignment and behavioral arbitration. In naturalistic environments, causal relationships between actions and their consequences are often hidden from direct observation. To overcome this structural credit-assignment problem, agents have to infer causal structures from experience. Here, we developed a computational model which formalizes action selection as the competition between structural representations, while action control is arbitrated towards the representation that minimizes surprise over time. To validate this model, we recorded behavioral and electrophysiological data from human participants in a novel task in which independent decisions are followed by outcomes, whereby the decision-outcome mapping is unknown. The model could account for patterns of choice behavior revealing clear hallmarks of credit assignment. Model-based analysis of EEG activity confirmed central model characteristics of concurrent prediction errors and a signature of evidence accumulation and behavioral arbitration. These findings highlight a key role of surprise minimization for both value and representation learning and reveal neural correlates of credit assignment.

Agency rescues competition for credit assignment among predictive cues from adverse learning conditions

Neural mechanisms of credit assignment for delayed outcomes during contingent learning

Plasticity of Neuronal Dynamics in the Lateral Habenula for Cue-Punishment Associative Learning.

Prediction Error During Retrospective Revaluation of Causal Associations in Humans: Fmri Evidence in Favor of an Associative Model of Learning

A causal link between prediction errors, dopamine neurons and learning

Neural activity ramps in frontal cortex signal extended motivation during learning

Surprise-minimization as a solution to the structural credit assignment problem

A computational model of behavioral adaptation to solve the credit assignment problem

The Dopamine Prediction Error: Contributions to Associative Models of Reward Learning

Parallel Contributions of Distinct Human Memory Systems During Probabilistic Learning.

The Role of the Lateral Frontal Cortex in Causal Associative Learning: Exploring Preventative and Super-Learning.

Selective Credit Assignment

Dynamic shaping of dopamine signals during probabilistic Pavlovian conditioning

Action prediction error: a value-free dopaminergic teaching signal that drives stable learning

Dopamine Release Plateau and Outcome Signals in Dorsal Striatum Contrast with Classic Reinforcement Learning Formulations

Survival of the salient: Aversive learning rescues otherwise forgettable memories via neural reactivation and post-encoding hippocampal connectivity

Credit Assignment: Challenges and Opportunities in Developing Human-like AI Agents

Mesostriatal dopamine is sensitive to changes in specific cue-reward contingencies

Agency Modulates the Lateral and Medial Prefrontal Cortex Responses in Belief-Based Decision Making

Dissociable Contributions of Basolateral Amygdala and Ventrolateral Orbitofrontal Cortex to Flexible Learning Under Uncertainty