Abstract:This study introduces a novel framework for understanding the cognitive underpinnings of individual behavior which often deviates from rational decision-making aimed at maximizing rewards in real-life scenarios. We propose a structure learning approach to infer an agent's internal model, composed of a learning rule and an internal representation of the environment. Crucially, the combined contribution of these components - rather than their individual contribution - determines the overall optimality of the agent's behavior. By exploring various combinations of learning rules and environment representations, we identify the most probable agent model structure for each individual. We apply this framework to analyze rats' learning behavior in a free-choice task within a T-maze with return arms, evaluating different internal models based on optimal and suboptimal learning rules, along with multiple possible representations of the T-maze decision graph. Identifying the most likely agent model structure based on the rats' behavioral data reveals that slower learning rats employed either a suboptimal or a moderately optimal agent model, whereas fast learners employ an optimal agent model. Using the inferred agent models for each rat, we explore the qualitative differences in their individual learning processes. Policy entropy, derived from the inferred agent models, also highlights variations in the balance between exploration and exploitation strategies among the rats. Traditional reinforcement learning approaches to addressing suboptimal behavior focus separately on either suboptimal learning rules or flawed environment representations. Our approach jointly models these components, revealing that suboptimal decisions can arise from complex interactions between learning rules and environment representations within an agent's internal model. This provides deeper insights into the cognitive mechanisms underlying real-world decision-making.

Belief dynamics extraction

Modeling Complex Animal Behavior with Latent State Inverse Reinforcement Learning

Unveiling the latent dynamics in social cognition with multi-agent inverse reinforcement learning

Where Do You Think You're Going?: Inferring Beliefs about Dynamics from Behavior

Dynamical latent state computation in the male macaque posterior parietal cortex

Meta-Dynamical State Space Models for Integrative Neural Data Analysis

Probabilistic models of individual and collective animal behavior

Decoding Neural Activity to Assess Individual Latent State in Ecologically Valid Contexts

Parsing neural dynamics with infinite recurrent switching linear dynamical systems

Capturing Optimal and Suboptimal behavior Of Agents Via Structure Learning Of Their Internal Model

Discovering Latent States for Model Learning: Applying Sensorimotor Contingencies Theory and Predictive Processing to Model Context

Collective behavior from surprise minimization

Exploring Behavior-Relevant and Disentangled Neural Dynamics with Generative Diffusion Models

Learning an Internal Dynamics Model from Control Demonstration

Active inference on discrete state-spaces: a synthesis

Dynamic reinforcement learning reveals time-dependent shifts in strategy during reward learning.

Modeling sensory-motor decisions in natural behavior

SIMPL: Scalable and hassle-free optimization of neural representations from behaviour

Thoughtful faces: inferring internal states across species using facial features

Neural Foundations of Mental Simulation: Future Prediction of Latent Representations on Dynamic Scenes

Active Inference as a Model of Agency