Abstract:In this work, we address a relatively unexplored aspect of designing agents that learn fromhuman reward.We investigate howan agent’s non-task behavior can affect a human trainer’s training and agent learning. We use the TAMER framework, which facilitates the training of agents by human-generated reward signals, i.e., judgements of the quality of the agent’s actions, as the foundation for our investigation. Then, starting from the premise that the interaction between the agent and the trainer should be bi-directional, we propose two new training interfaces to increase a human trainer’s active involvement in the training process and thereby improve the agent’s task performance. One provides information on the agent’s uncertainty which is a metric calculated as data coverage, the other on its performance. Our results from a 51-subject user study show that these interfaces can induce the trainers to train longer and give more feedback. The agent’s performance, however, increases only in response to the addition of performance-oriented information, not by sharing uncertainty levels. These results suggest that the organizational maxim about human behavior, “you get what you measure”—i.e., sharing metrics with people causes them to focus on optimizing those metrics while de-emphasizing other objectives—also applies to the training of agents. Using principle component analysis, we show how trainers in the two conditions train agents differently. In addition, by simulating the influence of the agent’s uncertainty–informative B Guangliang Li g.li@uva.nl Shimon Whiteson shimon.whiteson@cs.ox.ac.uk W. Bradley Knox bradknox@mit.edu Hayley Hung h.hung@tudelft.nl 1 University of Amsterdam, Amsterdam, The Netherlands 2 University of Oxford, Oxford, UK 3 Massachusetts Institute of Technology, Cambridge, USA 4 Technical University of Delft, Delft, The Netherlands

Towards Learning from Implicit Human Reward: (extended Abstract)

A Large-Scale Study of Agents Learning from Human Reward

Facial feedback for reinforcement learning: a case study and offline analysis using the TAMER framework

Using Informative Behavior to Increase Engagement While Learning from Human Reward.

Human Social Feedback for Efficient Interactive Reinforcement Agent Learning

UvA-DARE ( Digital Academic Repository ) Using Informative Behavior to Increase Engagement While Learning from Human Reward

Learning from Human Reward Benefits from Socio-Competitive Feedback

Using informative behavior to increase engagement in the tamer framework

Social Interaction for Efficient Agent Learning from Human Reward.

Leveraging Social Networks to Motivate Humans to Train Agents (extended Abstract)

Reinforcement Learning from Demonstration and Human Reward

Learning Human Rewards by Inferring Their Latent Intelligence Levels in Multi-Agent Games: A Theory-of-Mind Approach with Application to Driving Data

Leveraging social networks to motivate humans to train agents.

Improving Interactive Reinforcement Agent Planning with Human Demonstration

Adaptively Shaping Reinforcement Learning Agents Via Human Reward

DQN-TAMER: Human-in-the-Loop Reinforcement Learning with Intractable Feedback

Accelerating the Learning of TAMER with Counterfactual Explanations

Modeling Affect-based Intrinsic Rewards for Exploration and Learning

Face valuing: Training user interfaces with facial expressions and reinforcement learning

FRESH: Interactive Reward Shaping in High-Dimensional State Spaces using Human Feedback

Accelerating Reinforcement Learning Agent with EEG-based Implicit Human Feedback