Abstract:The structural credit assignment problem arises when the causal structure between actions and subsequent outcomes is hidden from direct observation. To solve this problem and enable goal-directed behavior, an agent has to infer structure and form a representation thereof. In the scope of this study, we investigate a possible solution in the human brain. We recorded behavioral and electrophysiological data from human participants in a novel variant of the bandit task, where multiple actions lead to multiple outcomes. Crucially, the mapping between actions and outcomes was hidden and not instructed to the participants. Human choice behavior revealed clear hallmarks of credit assignment and learning. Moreover, a computational model which formalizes action selection as the competition between multiple representations of the hidden structure was fit to account for participants data. Starting in a state of uncertainty about the correct representation, the central mechanism of this model is the arbitration of action control towards the representation which minimizes surprise about outcomes. Crucially, single-trial latent-variable analysis reveals that the neural patterns clearly support central quantitative predictions of this surprise minimization model. The results suggest that neural activity is not only related to reinforcement learning under correct as well as incorrect task representations but also reflects central mechanisms of credit assignment and behavioral arbitration. In naturalistic environments, causal relationships between actions and their consequences are often hidden from direct observation. To overcome this structural credit-assignment problem, agents have to infer causal structures from experience. Here, we developed a computational model which formalizes action selection as the competition between structural representations, while action control is arbitrated towards the representation that minimizes surprise over time. To validate this model, we recorded behavioral and electrophysiological data from human participants in a novel task in which independent decisions are followed by outcomes, whereby the decision-outcome mapping is unknown. The model could account for patterns of choice behavior revealing clear hallmarks of credit assignment. Model-based analysis of EEG activity confirmed central model characteristics of concurrent prediction errors and a signature of evidence accumulation and behavioral arbitration. These findings highlight a key role of surprise minimization for both value and representation learning and reveal neural correlates of credit assignment.

Capturing Optimal and Suboptimal behavior Of Agents Via Structure Learning Of Their Internal Model

Inferring Time-Varying Internal Models of Agents Through Dynamic Structure Learning

Learning the structure of the world: The adaptive nature of state-space and action representations in multi-stage decision-making

Modeling sensory-motor decisions in natural behavior

Humans rationally balance detailed and temporally abstract world models

Learning Optimal Behavior Through Reasoning and Experiences

Exploring and Learning Structure: Active Inference Approach in Navigational Agents

An Active Inference Approach to Modeling Structure Learning: Concept Learning as an Example Case

Behavior Decision of Mobile Robot With a Neurophysiologically Motivated Reinforcement Learning Model

External Model Motivated Agents: Reinforcement Learning for Enhanced Environment Sampling

Simulating how animals learn: a new modelling framework applied to the process of optimal foraging

Toward a Psychology of Deep Reinforcement Learning Agents Using a Cognitive Architecture

Surprise-minimization as a solution to the structural credit assignment problem

Structure Learning in Motor Control:A Deep Reinforcement Learning Model

Belief dynamics extraction

A Cognitive Modeling Approach to Strategy Formation in Dynamic Decision Making

Unveiling the latent dynamics in social cognition with multi-agent inverse reinforcement learning

Reward is not Necessary: How to Create a Modular & Compositional Self-Preserving Agent for Life-Long Learning

Simple Plans or Sophisticated Habits? State, Transition and Learning Interactions in the Two-Step Task

How Do You Act? An Empirical Study to Understand Behavior of Deep Reinforcement Learning Agents

Where Do You Think You're Going?: Inferring Beliefs about Dynamics from Behavior