Abstract:This study introduces a novel framework for understanding the cognitive underpinnings of individual behavior which often deviates from rational decision-making aimed at maximizing rewards in real-life scenarios. We propose a structure learning approach to infer an agent's internal model, composed of a learning rule and an internal representation of the environment. Crucially, the combined contribution of these components - rather than their individual contribution - determines the overall optimality of the agent's behavior. By exploring various combinations of learning rules and environment representations, we identify the most probable agent model structure for each individual. We apply this framework to analyze rats' learning behavior in a free-choice task within a T-maze with return arms, evaluating different internal models based on optimal and suboptimal learning rules, along with multiple possible representations of the T-maze decision graph. Identifying the most likely agent model structure based on the rats' behavioral data reveals that slower learning rats employed either a suboptimal or a moderately optimal agent model, whereas fast learners employ an optimal agent model. Using the inferred agent models for each rat, we explore the qualitative differences in their individual learning processes. Policy entropy, derived from the inferred agent models, also highlights variations in the balance between exploration and exploitation strategies among the rats. Traditional reinforcement learning approaches to addressing suboptimal behavior focus separately on either suboptimal learning rules or flawed environment representations. Our approach jointly models these components, revealing that suboptimal decisions can arise from complex interactions between learning rules and environment representations within an agent's internal model. This provides deeper insights into the cognitive mechanisms underlying real-world decision-making.

External Model Motivated Agents: Reinforcement Learning for Enhanced Environment Sampling

Shattering the Agent-Environment Interface for Fine-Tuning Inclusive Language Models

Learning Off-policy with Model-based Intrinsic Motivation For Active Online Exploration

A conceptual framework for externally-influenced agents: an assisted reinforcement learning review

Model-based Exploration Strategy to Accelerate Deterministic Strategy Algorithm Training

Surprise-Adaptive Intrinsic Motivation for Unsupervised Reinforcement Learning

Learning to Adapt in Dynamic, Real-World Environments Through Meta-Reinforcement Learning

World Models Increase Autonomy in Reinforcement Learning

Inferring Time-Varying Internal Models of Agents Through Dynamic Structure Learning

Importance Sampling-Guided Meta-Training for Intelligent Agents in Highly Interactive Environments

Experimental Evidence that Empowerment May Drive Exploration in Sparse-Reward Environments

Discovering Latent States for Model Learning: Applying Sensorimotor Contingencies Theory and Predictive Processing to Model Context

Maximum Entropy Model-based Reinforcement Learning

Multi-Agent Reinforcement Learning Via Directed Exploration Method

Curiosity & Entropy Driven Unsupervised RL in Multiple Environments

Environment as Policy: Learning to Race in Unseen Tracks

Adaptive action supervision in reinforcement learning from real-world multi-agent demonstrations

Reinforcement Learning for Sparse-Reward Object-Interaction Tasks in a First-person Simulated 3D Environment

Capturing Optimal and Suboptimal behavior Of Agents Via Structure Learning Of Their Internal Model

Improving Reinforcement Learning Efficiency with Auxiliary Tasks in Non-Visual Environments: A Comparison