Abstract:Novelty is a double-edged sword for agents and animals alike: they might benefit from untapped resources or face unexpected costs or dangers such as predation. The conventional exploration/exploitation tradeoff is thus coloured by risk-sensitivity. A wealth of experiments has shown how animals solve this dilemma, for example using intermittent approach. However, there are large individual differences in the nature of approach, and modeling has yet to elucidate how this might be based on animals' differing prior expectations about reward and threat and degrees of risk aversion. To capture these factors, we built a Bayes adaptive Markov decision process model with three key components: an adaptive hazard function capturing potential predation, an intrinsic reward function providing the urge to explore, and a conditional value at risk (CVaR) objective, which is a contemporary measure of trait risk-sensitivity. We fit this model to a coarse-grain abstraction of the behaviour of 26 animals who freely explored a novel object in an open-field arena (Akiti et al. 110, 2022). We show that the model captures both quantitative (frequency, duration of exploratory bouts) and qualitative (stereotyped tail-behind) features of behavior, including the substantial idiosyncrasies that were observed. We find that "brave" animals, though varied in their behavior, generally are more risk neutral, and enjoy a flexible hazard prior. They begin with cautious exploration, and quickly transition to confident approach to maximize exploration for reward. On the other hand, "timid" animals, characterized by risk aversion and high and inflexible hazard priors, display self-censoring that leads to the sort of asymptotic maladaptive behavior that is often associated with psychiatric illnesses such as anxiety and depression. Explaining risk-sensitive exploration using factorized parameters of reinforcement learning models could aid in the understanding, diagnosis, and treatment of psychiatric abnormalities in humans and other animals.

A Behavioral Model for Exploration vs. Exploitation: Theoretical Framework and Experimental Evidence

Humans adaptively resolve the explore-exploit dilemma under cognitive constraints: Evidence from a multi-armed bandit task

Learning and choosing in an uncertain world: An investigation of the explore-exploit dilemma in static and dynamic environments

Exploration Unbound

Risking your Tail: Modeling Individual Differences in Risk-sensitive Exploration using Bayes Adaptive Markov Decision Processes

Uncertainty quantification and exploration-exploitation trade-off in humans

The Exploration-Exploitation Dilemma Revisited: An Entropy Perspective

Non-strategic Econometrics (for Initial Play)

Reward Uncertainty for Exploration in Preference-based Reinforcement Learning

Incentivizing Exploration with Heterogeneous Value of Money

Decision from Models: Generalizing Probability Information to Novel Tasks.

Incentivized Exploration for Multi-Armed Bandits under Reward Drift.

Alliance Choices: A Behavioral Theory of Problemistic and Slack Searches under CEO Overconfidence

Leveraging Transition Exploratory Bonus for Efficient Exploration in Hard-Transiting Reinforcement Learning Problems.

Toward a Coherent Understanding of Risky, Intertemporal, and Spatial Choices: Evidence from Eye-Tracking and Subjective Evaluation

Modeling sensory-motor decisions in natural behavior

Social Learning and the Exploration-Exploitation Tradeoff

Regulatory fit and systematic exploration in a dynamic decision-making environment.

The Perils of Exploration under Competition: A Computational Modeling Approach

Efficient Exploration in Resource-Restricted Reinforcement Learning

Model-based Exploration Strategy to Accelerate Deterministic Strategy Algorithm Training