Abstract:In this work, we propose a video-based transfer learning approach for predicting problem outcomes of students working with an intelligent tutoring system (ITS). By analyzing a student's face and gestures, our method predicts the outcome of a student answering a problem in an ITS from a video feed. Our work is motivated by the reasoning that the ability to predict such outcomes enables tutoring systems to adjust interventions, such as hints and encouragement, and to ultimately yield improved student learning. We collected a large labeled dataset of student interactions with an intelligent online math tutor consisting of 68 sessions, where 54 individual students solved 2,749 problems. The dataset is public and available at <a class="link-external link-https" href="https://www.cs.bu.edu/faculty/betke/research/learning/" rel="external noopener nofollow">this https URL</a> . Working with this dataset, our transfer-learning challenge was to design a representation in the source domain of pictures obtained "in the wild" for the task of facial expression analysis, and transferring this learned representation to the task of human behavior prediction in the domain of webcam videos of students in a classroom environment. We developed a novel facial affect representation and a user-personalized training scheme that unlocks the potential of this representation. We designed several variants of a recurrent neural network that models the temporal structure of video sequences of students solving math problems. Our final model, named ATL-BP for Affect Transfer Learning for Behavior Prediction, achieves a relative increase in mean F-score of 50% over the state-of-the-art method on this new dataset.

Visual Interest Prediction with Attentive Multi-Task Transfer Learning

Deep Vision Multimodal Learning: Methodology, Benchmark, and Trend

Transfer Vision Patterns for Multi-Task Pixel Learning.

A Visual Analytical Approach for Transfer Learning in Classification.

Beyond Vision: A Multimodal Recurrent Attention Convolutional Neural Network for Unified Image Aesthetic Prediction Tasks

Multi-Output Learning Based on Multimodal GCN and Co-Attention for Image Aesthetics and Emotion Analysis

Joint Learning for Face Alignment and Face Transfer with Depth Image

Multi-Task Convolutional Neural Network for Image Aesthetic Assessment

Utilizing Deep Learning Towards Multi-modal Bio-sensing and Vision-based Affective Computing

Dynamic Bayesian Network Based Visual Focus of Attention Recognition

Leveraging Affect Transfer Learning for Behavior Prediction in an Intelligent Tutoring System

Correlated Attribute Transfer with Multi-Task Graph-Guided Fusion.

Probabilistic Multi-Task Learning for Visual Saliency Estimation in Video

Multimodal Contrastive Training for Visual Representation Learning

Visual sentiment analysis using data-augmented deep transfer learning techniques

Deep Additive Least Squares Support Vector Machines for Classification with Model Transfer

Adaptive and Dynamic Knowledge Transfer in Multi-task Learning with Attention Networks.

Simultaneous face detection and 360 degree headpose estimation

Multi-Branch Network for Imagery Emotion Prediction

On the Surprising Effectiveness of Attention Transfer for Vision Transformers

Human Visual Attention Prediction Boosts Learning & Performance of Autonomous Driving Agents