Abstract:Generative AI, particularly Language Models (LMs), has the potential to transform real-world domains with societal impact, particularly where access to experts is limited. For example, in education, training novice educators with expert guidance is important for effectiveness but expensive, creating significant barriers to improving education quality at scale. This challenge disproportionately harms students from under-served communities, who stand to gain the most from high-quality education. We introduce Tutor CoPilot, a novel Human-AI approach that leverages a model of expert thinking to provide expert-like guidance to tutors as they tutor. This study is the first randomized controlled trial of a Human-AI system in live tutoring, involving 900 tutors and 1,800 K-12 students from historically under-served communities. Following a preregistered analysis plan, we find that students working with tutors that have access to Tutor CoPilot are 4 percentage points (p.p.) more likely to master topics (p<0.01). Notably, students of lower-rated tutors experienced the greatest benefit, improving mastery by 9 p.p. We find that Tutor CoPilot costs only $20 per-tutor annually. We analyze 550,000+ messages using classifiers to identify pedagogical strategies, and find that tutors with access to Tutor CoPilot are more likely to use high-quality strategies to foster student understanding (e.g., asking guiding questions) and less likely to give away the answer to the student. Tutor interviews highlight how Tutor CoPilot's guidance helps tutors to respond to student needs, though they flag issues in Tutor CoPilot, such as generating suggestions that are not grade-level appropriate. Altogether, our study of Tutor CoPilot demonstrates how Human-AI systems can scale expertise in real-world domains, bridge gaps in skills and create a future where high-quality education is accessible to all students.

Reinforcement learning tutor better supported lower performers in a math task

Leveraging Deep Reinforcement Learning for Metacognitive Interventions across Intelligent Tutoring Systems

RLTutor: Reinforcement Learning Based Adaptive Tutoring System by Modeling Virtual Student with Fewer Interactions

A Deep Reinforcement Learning Agent for Geometry Online Tutoring

Using Large Language Models to Assess Tutors' Performance in Reacting to Students Making Math Errors

MathDQN: Solving Arithmetic Word Problems Via Deep Reinforcement Learning.

Using Machine Teaching to Investigate Human Assumptions when Teaching Reinforcement Learners

Improving Student Learning with Hybrid Human-AI Tutoring: A Three-Study Quasi-Experimental Investigation

Do Tutors Learn from Equity Training and Can Generative AI Assess It?

Improving Teaching at Scale: Can AI Be Incorporated Into Professional Development to Create Interactive, Personalized Learning for Teachers?

Aligning Tutor Discourse Supporting Rigorous Thinking with Tutee Content Mastery for Predicting Math Achievement

Revealing Networks: Understanding Effective Teacher Practices in AI-Supported Classrooms using Transmodal Ordered Network Analysis

Empowering Private Tutoring by Chaining Large Language Models

Implementing Learning Principles with a Personal AI Tutor: A Case Study

Better than Your Teacher: LLM Agents that learn from Privileged AI Feedback

Towards the Pedagogical Steering of Large Language Models for Tutoring: A Case Study with Modeling Productive Failure

Improving Assessment of Tutoring Practices using Retrieval-Augmented Generation

Human Decision Makings on Curriculum Reinforcement Learning with Difficulty Adjustment

Effective and Scalable Math Support: Evidence on the Impact of an AI- Tutor on Math Achievement in Ghana

Tutor CoPilot: A Human-AI Approach for Scaling Real-Time Expertise

Using Think-Aloud Data to Understand Relations between Self-Regulation Cycle Characteristics and Student Performance in Intelligent Tutoring Systems