Abstract:Adaptive systems teach and adapt to humans; their promise is to improve education by minimizingthe subset of items presented to students while maximizing student outcomes (Cen et al., 2007). In this context, items are questions, problems, or tasks that can be graded individually. The adaptive tutoring community has tacitly adopted conventions for evaluating tutoring systems (Dhanani et al., 2014) by using classification evaluation metrics that assess the student model component— student models are the subsystems that forecast whether a learner will answer the next item correctly. Unfortunately, it is not clear how intuitive classification metrics are for practitioners with little machine learning background. Moreover, our experiments on real and synthetic data reveal that it is possible to have student models that are very predictive (as measured by traditional classification metrics), yet provide little to no value to the learner. Additionally, when we compare alternative tutoring systems with classification metrics, we discover that they may favor tutoring systems that require higher student effort with no evidence that students are learning more. That is, when comparing two alternative systems, classification metrics may prefer a suboptimal system. We recently proposed Learner Effort-Outcomes Paradigm (Leopard) for automatic evaluation of adaptive tutoring (Gonz´alez-Brenes & Huang, 2015). Leopard extends on prior work on alternatives to classification evaluation metrics (Lee & Brunskill, 2012). At its core, Leopard quantifies both the effort and outcomes of students in adaptive tutoring. Even though these metrics are novel by itself, our contribution is approximating both without a randomized control trial. In this talk, we will describe our recently published results on meta-evaluating Leopard and conventional classification metrics. Additionally, we will present preliminary results of framing the value of an educational intervention as multi-objective programming. We argue that human-propelled machine learning, and educational technology in particular, aim to optimize the Pareto boundary of effort and outcomes of humans.

Improved Performances and Motivation in Intelligent Tutoring Systems: Combining Machine Learning and Learner Choice

Multi-Armed Bandits for Intelligent Tutoring Systems

Using Self-Determination Theory in Social Robots to Increase Motivation in L2 Word Learning

Iterative Teacher-Aware Learning

Using adaptive learning technologies to personalize instruction to student interests: The impact of relevant contexts on performance and learning outcomes.

Automated Personalized Feedback Improves Learning Gains in an Intelligent Tutoring System

Mastery-Oriented Shared Student/System Control Over Problem Selection in a Linear Equation Tutor

Leveraging Deep Reinforcement Learning for Metacognitive Interventions across Intelligent Tutoring Systems

Motivation and performance in a game-based intelligent tutoring system.

How Learner Control and Explainable Learning Analytics on Skill Mastery Shape Student Desires to Finish and Avoid Loss in Tutored Practice

SKOPE-IT (Shareable Knowledge Objects as Portable Intelligent Tutors): overlaying natural language tutoring on an adaptive learning system for mathematics

Design and Implementation of an Intelligent Tutoring System in the View of Learner Autonomy

Towards Goal-oriented Intelligent Tutoring Systems in Online Education

Utility-based Adaptive Teaching Strategies using Bayesian Theory of Mind

Reinforcement learning tutor better supported lower performers in a math task

Evaluating an Intelligent Tutoring System for Personalized Math Teaching

Using technology to make learning fun: technology use is best made fun and challenging to optimize intrinsic motivation and engagement

Getting too personal(ized): The importance of feature choice in online adaptive algorithms

Evaluating Alternative Training Interventions Using Personalized Computational Models of Learning

The Leopard Framework: Towards Understanding Educational Technology Interventions with a Pareto Efficiency Perspective

For Educational Inclusiveness: Design and Implementation of an Intelligent Tutoring System for Student-Athletes Based on Self-Determination Theory