Tutorly: Turning Programming Videos Into Apprenticeship Learning Environments with LLMs

Wengxi Li,Roy Pea,Nick Haber,Hari Subramonyam

2024-05-22

Abstract:Online programming videos, including tutorials and streamcasts, are widely popular and contain a wealth of expert knowledge. However, effectively utilizing these resources to achieve targeted learning goals can be challenging. Unlike direct tutoring, video content lacks tailored guidance based on individual learning paces, personalized feedback, and interactive engagement necessary for support and monitoring. Our work transforms programming videos into one-on-one tutoring experiences using the cognitive apprenticeship framework. Tutorly, developed as a JupyterLab Plugin, allows learners to (1) set personalized learning goals, (2) engage in learning-by-doing through a conversational LLM-based mentor agent, (3) receive guidance and feedback based on a student model that steers the mentor moves. In a within-subject study with 16 participants learning exploratory data analysis from a streamcast, Tutorly significantly improved their performance from 61.9% to 76.6% based on a post-test questionnaire. Tutorly demonstrates the potential for enhancing programming video learning experiences with LLM and learner modeling.

Human-Computer Interaction,Machine Learning

What problem does this paper attempt to address?

### Problems the Paper Aims to Solve

The paper aims to address the issue that online programming video tutorials, while rich in expert knowledge, lack personalized guidance, instant feedback, and interactive engagement. Specifically, the paper proposes a system called Tutorly, which enhances the learning experience by transforming programming videos into one-on-one tutoring sessions. Tutorly, as a JupyterLab plugin, leverages the Cognitive Apprenticeship Framework and the capabilities of large language models (LLMs) to provide personalized guidance and instant feedback to learners.

#### Main Objectives
1. **Personalized Learning Goals**: Enable learners to set learning goals based on their individual needs.
2. **Hands-on Learning**: Achieve learning by doing through interaction with an LLM-based conversational tutor agent.
3. **Guidance Based on Student Models**: Provide guidance and feedback based on student models, making tutoring more targeted.
4. **Improving Learning Outcomes**: Through experimental validation, demonstrate that learners' performance significantly improves after using Tutorly.

The paper showcases through empirical research the potential of Tutorly in enhancing the learning experience of programming videos and proves the effectiveness of this approach.