Multimodal Machine Learning for Automated Assessment of Attention-Related Processes during Learning

Babette Bühler

2024-07-08

Abstract:Attention is a key factor for successful learning, with research indicating strong associations between (in)attention and learning outcomes. This dissertation advanced the field by focusing on the automated detection of attention-related processes using eye tracking, computer vision, and machine learning, offering a more objective, continuous, and scalable assessment than traditional methods such as self-reports or observations. It introduced novel computational approaches for assessing various dimensions of (in)attention in online and classroom learning settings and addressing the challenges of precise fine-granular assessment, generalizability, and in-the-wild data quality. First, this dissertation explored the automated detection of mind-wandering, a shift in attention away from the learning task. Aware and unaware mind wandering were distinguished employing a novel multimodal approach that integrated eye tracking, video, and physiological data. Further, the generalizability of scalable webcam-based detection across diverse tasks, settings, and target groups was examined. Second, this thesis investigated attention indicators during online learning. Eye-tracking analyses revealed significantly greater gaze synchronization among attentive learners. Third, it addressed attention-related processes in classroom learning by detecting hand-raising as an indicator of behavioral engagement using a novel view-invariant and occlusion-robust skeleton-based approach. This thesis advanced the automated assessment of attention-related processes within educational settings by developing and refining methods for detecting mind wandering, on-task behavior, and behavioral engagement. It bridges educational theory with advanced methods from computer science, enhancing our understanding of attention-related processes that significantly impact learning outcomes and educational practices.

Human-Computer Interaction

What problem does this paper attempt to address?

### What problem does this paper attempt to solve?

This paper primarily focuses on methods for the automated assessment of attention-related processes during learning. Specifically, it leverages eye-tracking, computer vision, and machine learning technologies to propose a more objective, continuous, and scalable assessment method compared to traditional approaches such as self-reporting or observation. The main research content of the paper includes the following aspects:

1. **Mind Wandering Detection**:
- Investigates the automatic detection of mind wandering (attention shifting away from the learning task).
- Differentiates between conscious and unconscious mind wandering and improves the detection accuracy of these two types through predictive modeling based on eye movement data.
- Proposes a new multimodal approach that integrates eye-tracking, video, and physiological data to enhance detection accuracy and robustness.
- Explores the generalizability of webcam-based mind wandering detection methods across different tasks, environments, and target groups.

2. **Attention Metrics in Online Learning**:
- Analyzes the phenomenon of attention synchronization during online learning.
- Finds significant eye movement synchronization among learners who are focused.

3. **Behavioral Engagement in Classroom Learning**:
- Utilizes a novel view-invariant and occlusion-robust skeletal-based method to detect hand-raising behavior in the classroom as an indicator of behavioral engagement.
- Explores the correlation between automatically labeled hand-raising behavior and learners' self-reported engagement, interest, and involvement, demonstrating the potential of large-scale video analysis.

In summary, this paper enhances the automated assessment of attention-related processes in educational environments by developing and improving methods for detecting mind wandering, task behavior, and behavioral engagement. It combines educational theory with advanced computer science technologies, deepening the understanding of attention-related processes that impact learning outcomes and educational practices.