Abstract:Is it possible for machines to think like humans? And if it is, how should we go about teaching them to do so? As early as 1950, Alan Turing stated that we ought to teach machines in the way of teaching a child. Reinforcement learning with human feedback (RLHF) has emerged as a strong candidate toward allowing agents to learn from human feedback in a naturalistic manner. RLHF is distinct from traditional reinforcement learning as it provides feedback from a human teacher in addition to a reward signal. It has been catapulted into public view by multiple high-profile AI applications, including OpenAI's ChatGPT, DeepMind's Sparrow, and Anthropic's Claude. These highly capable chatbots are already overturning our understanding of how AI interacts with humanity. The wide applicability and burgeoning success of RLHF strongly motivate the need to evaluate its social impacts. In light of recent developments, this paper considers an important question: can RLHF be developed and used without negatively affecting human societies? Our objectives are threefold: to provide a systematic study of the social effects of RLHF; to identify key social and ethical issues of RLHF; and to discuss social impacts for stakeholders. Although text-based applications of RLHF have received much attention, it is crucial to consider when evaluating its social implications the diverse range of areas to which it may be deployed. We describe seven primary ways in which RLHF-based technologies will affect society by positively transforming human experiences with AI. This paper ultimately proposes that RLHF has potential to net positively impact areas of misinformation, AI value-alignment, bias, AI access, cross-cultural dialogue, industry, and workforce. As RLHF raises concerns that echo those of existing AI technologies, it will be important for all to be aware and intentional in the adoption of RLHF.

A Review on Interactive Reinforcement Learning from Human Social Feedback

Reinforcement Learning based Embodied Agents Modelling Human Users Through Interaction and Multi-Sensory Perception

Mapping out the Space of Human Feedback for Reinforcement Learning: A Conceptual Framework

A Survey of Reinforcement Learning from Human Feedback

Reinforcement Learning Approaches in Social Robotics

Multi-trainer binary feedback interactive reinforcement learning

Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedback

Actor-Critic Reinforcement Learning with Simultaneous Human Control and Feedback

Human-Robot Emotional Interaction Model Based on Reinforcement Learning

Perspectives on the Social Impacts of Reinforcement Learning with Human Feedback

Towards Interactive Reinforcement Learning with Intrinsic Feedback

Bootstrapping Adaptive Human-Machine Interfaces with Offline Reinforcement Learning

RLHF-Blender: A Configurable Interactive Interface for Learning from Diverse Human Feedback

A Survey On Enhancing Reinforcement Learning in Complex Environments: Insights from Human and LLM Feedback

Interactive Dialogue Agents via Reinforcement Learning on Hindsight Regenerations

Facial feedback for reinforcement learning: a case study and offline analysis using the TAMER framework

PEBBLE: Feedback-Efficient Interactive Reinforcement Learning via Relabeling Experience and Unsupervised Pre-training

Accelerating Reinforcement Learning using EEG-based implicit human feedback

GUIDE: Real-Time Human-Shaped Agents

A Survey on Interactive Reinforcement Learning: Design Principles and Open Challenges

Review of the Progress of Communication-Based Multi-Agent Reinforcement Learning