Abstract:Reinforcement learning (RL) models usually assume a stationary internal model structure of agents, which consists of fixed learning rules and environment representations. However, this assumption does not allow accounting for real problem solving by individuals who can exhibit irrational behaviors or hold inaccurate beliefs about their environment. In this work, we present a novel framework called Dynamic Structure Learning (DSL), which allows agents to adapt their learning rules and internal representations dynamically. This structural flexibility enables a deeper understanding of how individuals learn and adapt in real-world scenarios. The DSL framework reconstructs the most likely sequence of agent structures - sourced from a pool of learning rules and environment models - based on observed behaviors. The method provides insights into how an agent's internal structure model evolves as it transitions between different structures throughout the learning process. We applied our framework to study rat behavior in a maze task. Our results demonstrate that rats progressively refine their mental map of the maze, evolving from a suboptimal representation associated with repetitive errors to an optimal one that guides efficient navigation. Concurrently, their learning rules transition from heuristic-based to more rational approaches. These findings underscore the importance of both credit assignment and representation learning in complex behaviors. By going beyond simple reward-based associations, our research offers valuable insights into the cognitive mechanisms underlying decision-making in natural intelligence. DSL framework allows better understanding and modeling how individuals in real-world scenarios exhibit a level of adaptability that current AI systems have yet to achieve.

Learning Dynamic Abstract Representations for Sample-Efficient Reinforcement Learning

Graph learning-based generation of abstractions for reinforcement learning

Learning Markov State Abstractions for Deep Reinforcement Learning

State Abstraction via Deep Supervised Hash Learning

Sample-efficient multi-agent reinforcement learning with masked reconstruction

Building Minimal and Reusable Causal State Abstractions for Reinforcement Learning

Dynamic Sparse Training for Deep Reinforcement Learning

Sample Efficient Deep Reinforcement Learning with Online State Abstraction and Causal Transformer Model Prediction

Learning Discrete State Abstractions With Deep Variational Inference

Relational Abstractions for Generalized Reinforcement Learning on Symbolic Problems

Learning on Abstract Domains: A New Approach for Verifiable Guarantee in Reinforcement Learning

Autonomous Option Invention for Continual Hierarchical Reinforcement Learning and Planning

Dynamics-aware Embeddings

Contrastive Abstraction for Reinforcement Learning

DEAR: Disentangled Environment and Agent Representations for Reinforcement Learning without Reconstruction

Inferring Time-Varying Internal Models of Agents Through Dynamic Structure Learning

A Theory of Abstraction in Reinforcement Learning

Generative Adversarial Imagination for Sample Efficient Deep Reinforcement Learning

Learning Parsimonious Dynamics for Generalization in Reinforcement Learning

Learning Action Representations for Reinforcement Learning