Abstract:When inferring reward functions from human behavior (be it demonstrations, comparisons, physical corrections, or e-stops), it has proven useful to model the human as making noisy-rational choices, with a "rationality coefficient" capturing how much noise or entropy we expect to see in the human behavior. Prior work typically sets the rationality level to a constant value, regardless of the type, or quality, of human feedback. However, in many settings, giving one type of feedback (e.g. a demonstration) may be much more difficult than a different type of feedback (e.g. answering a comparison query). Thus, we expect to see more or less noise depending on the type of human feedback. In this work, we advocate that grounding the rationality coefficient in real data for each feedback type, rather than assuming a default value, has a significant positive effect on reward learning. We test this in both simulated experiments and in a user study with real human feedback. We find that overestimating human rationality can have dire effects on reward learning accuracy and regret. We also find that fitting the rationality coefficient to human data enables better reward learning, even when the human deviates significantly from the noisy-rational choice model due to systematic biases. Further, we find that the rationality level affects the informativeness of each feedback type: surprisingly, demonstrations are not always the most informative -- when the human acts very suboptimally, comparisons actually become more informative, even when the rationality level is the same for both. Ultimately, our results emphasize the importance and advantage of paying attention to the assumed human-rationality level, especially when agents actively learn from multiple types of human feedback.

Observable Changes of Hypotheses under Positive Reinforcement

Effectiveness of Positive Hypothesis Testing in Inductive and Deductive Rule Learning

Short-term plasticity as cause-effect hypothesis testing in distal reward learning

Hidden Reward: Affect and Its Prediction Errors as Windows Into Subjective Value

Theory of reinforcement schedules

Pavlovian learning and conditioned reinforcement

Behavioral and Neural Properties of Social Reinforcement Learning.

A theory of cerebral learning regulated by the reward system. I. Hypotheses and mathematical description

A bio-inspired reinforcement learning model that accounts for fast adaptation after punishment

Both probability and rate of reinforcement can affect the acquisition and maintenance of conditioned responses.

The Effect of Modeling Human Rationality Level on Learning Rewards from Multiple Feedback Types

Observational reinforcement learning in children and young adults

Affective learning: Awareness and aversion

How Instructed Knowledge Modulates the Neural Systems of Reward Learning

Learning in anticipation of reward and punishment: Perspectives across the human lifespan

Change point estimation by the mouse medial frontal cortex during probabilistic reward learning

Simple learning models can illuminate biased results from choice titration experiments

Manipulations of the context-response relationship reduce the expression of response habits

Dynamic reinforcement learning reveals time-dependent shifts in strategy during reward learning.

Learning Causal Overhypotheses through Exploration in Children and Computational Models

Evolving choice hysteresis in reinforcement learning: comparing the adaptive value of positivity bias and gradual perseveration