Abstract:Reinforcement learning (RL) models usually assume a stationary internal model structure of agents, which consists of fixed learning rules and environment representations. However, this assumption does not allow accounting for real problem solving by individuals who can exhibit irrational behaviors or hold inaccurate beliefs about their environment. In this work, we present a novel framework called Dynamic Structure Learning (DSL), which allows agents to adapt their learning rules and internal representations dynamically. This structural flexibility enables a deeper understanding of how individuals learn and adapt in real-world scenarios. The DSL framework reconstructs the most likely sequence of agent structures - sourced from a pool of learning rules and environment models - based on observed behaviors. The method provides insights into how an agent's internal structure model evolves as it transitions between different structures throughout the learning process. We applied our framework to study rat behavior in a maze task. Our results demonstrate that rats progressively refine their mental map of the maze, evolving from a suboptimal representation associated with repetitive errors to an optimal one that guides efficient navigation. Concurrently, their learning rules transition from heuristic-based to more rational approaches. These findings underscore the importance of both credit assignment and representation learning in complex behaviors. By going beyond simple reward-based associations, our research offers valuable insights into the cognitive mechanisms underlying decision-making in natural intelligence. DSL framework allows better understanding and modeling how individuals in real-world scenarios exhibit a level of adaptability that current AI systems have yet to achieve.

Effective Reinforcement Learning Based on Structural Information Principles

Effective Exploration Based on the Structural Information Principles

S2RL: Do We Really Need to Perceive All States in Deep Multi-Agent Reinforcement Learning?

S2rl

S2RL: DoWe Really Need to Perceive All States in Deep Multi-Agent Reinforcement Learning?

Hierarchical State Abstraction Based on Structural Information Principles

Effective and Stable Role-Based Multi-Agent Collaboration by Structural Information Principles

Structure in Deep Reinforcement Learning: A Survey and Open Problems

Hierarchical Reinforcement Learning Algorithm Based on Structural State-Space

An Improved Reinforcement Learning Method Based on Unsupervised Learning

Towards Efficient Long-Horizon Decision-Making Using Automated Structure Search Method of Hierarchical Reinforcement Learning for Edge Artificial Intelligence

Iteratively Learn Diverse Strategies with State Distance Information

Hybrid Information-driven Multi-agent Reinforcement Learning

On the Role of Information Structure in Reinforcement Learning for Partially-Observable Sequential Teams and Games

Rethinking Information Structures in RLHF: Reward Generalization from a Graph Theory Perspective

A Deep Reinforcement Learning Method for Structural Dominant Failure Modes Searching Based on Self-Play Strategy

Inferring Time-Varying Internal Models of Agents Through Dynamic Structure Learning

Symbolic Visual Reinforcement Learning: A Scalable Framework with Object-Level Abstraction and Differentiable Expression Search

A novel reinforcement learning-based method for structure optimization

SDRL: Interpretable and Data-efficient Deep Reinforcement Learning Leveraging Symbolic Planning

PROVABLY BENEFITS OF DEEP HIERARCHICAL RL