Abstract:Sensing and machine learning advances have enabled the unobtrusive measurement of physiological responses and facial expressions so as to estimate one's cognitive performance. This often boils down to mapping the states of the cognitive processes underpinning human cognition: physiological responses (e.g., heart rate) and facial expressions (e.g., frowning) often reflect the states of our cognitive processes. However, it remains unclear whether physiological responses and facial expressions used in one particular task (e.g., gaming) can reliably assess cognitive performance in another task (e.g., coding), because complex and diverse tasks often require varying levels and combinations of cognitive processes. In this paper, we measure the cross-task reliability of physiological and facial responses. Specifically, we assess cognitive performance based on physiological responses and facial expressions for 123 participants in 4 independent studies (3 studies for out-of-sampling training and testing, and 1 study for evaluation only): (1) a Pac-Man game, (2) an adaptive-assessment task, (3) a code-debugging task, and (4) a gaze-based game. We follow an ensemble learning approach after cross-training and cross-testing with all possible combinations of the 3 first datasets. We save the 4th dataset only for testing purposes, and we showcase how to engineer generalizable features that predict cognitive performance. Our results show that the extracted features do generalize, and can reliably predict cognitive performance across a diverse set of cognitive tasks that require different combinations of problem-solving, decision-making, and learning processes for their completion.

Piano Skills Assessment

Piano performance evaluation dataset with multilevel perceptual features

PianoMotion10M: Dataset and Benchmark for Hand Motion Generation in Piano Performance

Can Audio Reveal Music Performance Difficulty? Insights from the Piano Syllabus Dataset

Pianist Identification Using Convolutional Neural Networks

Combining piano performance dimensions for score difficulty classification

From Audio Encoders to Piano Judges: Benchmarking Performance Understanding for Solo Piano

RoboPianist: Dexterous Piano Playing with Deep Reinforcement Learning

PIAST: A Multimodal Piano Dataset with Audio, Symbolic and Text

Robust Piano Music Transcription Based on Computer Vision

Score Difficulty Analysis for Piano Performance Education based on Fingering

PiJAMA: Piano Jazz with Automatic MIDI Annotations

Piano4Play: An Automated Piano Transcription and Keyboard Visualization System using AI and Deep Learning Techniques

A 3D Guitar Fingering Assessing System Based on CNN-Hand Pose Estimation and SVR-Assessment

How does the teacher rate? Observations from the NeuroPiano dataset

Assessing Cognitive Performance Using Physiological and Facial Features

Automatic Assessment of Piano Performances Using Timbre and Pitch Features

Multi-Modal Machine Learning for Assessing Gaming Skills in Online Streaming: A Case Study with CS:GO

A Holistic Evaluation of Piano Sound Quality

Algorithms can Mimic Human Piano Performance: The Deep Blues of Music