Abstract:Artificial intelligence (AI) and especially reinforcement learning (RL) have the potential to enable agents to learn and perform tasks autonomously with superhuman performance. However, we consider RL as fundamentally a Human-in-the-Loop (HITL) paradigm, even when an agent eventually performs its task autonomously. In cases where the reward function is challenging or impossible to define, HITL approaches are considered particularly advantageous. The application of Reinforcement Learning from Human Feedback (RLHF) in systems such as ChatGPT demonstrates the effectiveness of optimizing for user experience and integrating their feedback into the training loop. In HITL RL, human input is integrated during the agent's learning process, allowing iterative updates and fine-tuning based on human feedback, thus enhancing the agent's performance. Since the human is an essential part of this process, we argue that human-centric approaches are the key to successful RL, a fact that has not been adequately considered in the existing literature. This paper aims to inform readers about current explainability methods in HITL RL. It also shows how the application of explainable AI (xAI) and specific improvements to existing explainability approaches can enable a better human-agent interaction in HITL RL for all types of users, whether for lay people, domain experts, or machine learning specialists. Accounting for the workflow in HITL RL and based on software and machine learning methodologies, this article identifies four phases for human involvement for creating HITL RL systems: (1) Agent Development, (2) Agent Learning, (3) Agent Evaluation, and (4) Agent Deployment. We highlight human involvement, explanation requirements, new challenges, and goals for each phase. We furthermore identify low-risk, high-return opportunities for explainability research in HITL RL and present long-term research goals to advance the field. Finally, we propose a vision of human-robot collaboration that allows both parties to reach their full potential and cooperate effectively.

Human-in-the-loop or AI-in-the-loop? Automate or Collaborate?

Human-in-the-loop machine learning: a state of the art

AI-in-the-Loop -- The impact of HMI in AI-based Application

Improving the Efficiency of Human-in-the-Loop Systems: Adding Artificial to Human Experts

The Human in the Infinite Loop: A Case Study on Revealing and Explaining Human-AI Interaction Loop Failures

Human-in-the-loop Techniques in Machine Learning.

A Survey of Human-in-the-loop for Machine Learning

Human-in-the-loop Machine Learning: A Macro-Micro Perspective

Human-in-the-Loop Reinforcement Learning: A Survey and Position on Requirements, Challenges, and Opportunities

Human-AI Collaboration in Real-World Complex Environment with Reinforcement Learning

Human-AI Collaboration in Decision-Making: Beyond Learning to Defer

Human-in-the-Loop Behavior Modeling via an Integral Concurrent Adaptive Inverse Reinforcement Learning

Learning Complementary Policies for Human-AI Teams

AI, Meet Human: Learning Paradigms for Hybrid Decision Making Systems

Human in the Loop for Machine Creativity

Keeping the organization in the loop: a socio-technical extension of human-centered artificial intelligence

What Are Humans Doing in the Loop? Co-Reasoning and Practical Judgment When Using Machine Learning-Driven Decision Aids

Evaluating Human-AI Collaboration: A Review and Methodological Framework

Human-In-The-Loop Machine Learning for Safe and Ethical Autonomous Vehicles: Principles, Challenges, and Opportunities

The Model Mastery Lifecycle: A Framework for Designing Human-AI Interaction

Human-Centric AI: Understanding and Enhancing Collaboration between Humans and Intelligent Systems