Abstract:We focus on data collection designs for the automated analysis of teacher-student interactions in live classrooms with the goal of identifying instructional activities (e.g., lecturing, discussion) and assessing the quality of dialogic instruction (e.g., analysis of questions). Our designs were motivated by multiple technical requirements and constraints. Most importantly, teachers could be individually micfied but their audio needed to be of excellent quality for automatic speech recognition (ASR) and spoken utterance segmentation. Individual students could not be micfied but classroom audio quality only needed to be sufficient to detect student spoken utterances. Visual information could only be recorded if students could not be identified. Design 1 used an omnidirectional laptop microphone to record both teacher and classroom audio and was quickly deemed unsuitable. In Designs 2 and 3, teachers wore a wireless Samson AirLine 77 vocal headset system, which is a unidirectional microphone with a cardioid pickup pattern. In Design 2, classroom audio was recorded with dual first- generation Microsoft Kinects placed at the front corners of the class. Design 3 used a Crown PZM-30D pressure zone microphone mounted on the blackboard to record classroom audio. Designs 2 and 3 were tested by recording audio in 38 live middle school classrooms from six U.S. schools while trained human coders simultaneously performed live coding of classroom discourse. Qualitative and quantitative analyses revealed that Design 3 was suitable for three of our core tasks: (1) ASR on teacher speech (word recognition rate of 66% and word overlap rate of 69% using Google Speech ASR engine); (2) teacher utterance segmentation (F-measure of 97%); and (3) student utterance segmentation (F-measure of 66%). Ideas to incorporate video and skeletal tracking with dual second-generation Kinects to produce Design 4 are discussed.

Audio-Based Group Detection for Classroom Dynamics Analysis.

An Automatic Detecting System Based on Sound Detection and Image Analysis

Multimodal Capture of Teacher-Student Interactions for Automated Dialogic Analysis in Live Classrooms

Person Detection in Collaborative Group Learning Environments Using Multiple Representations

Long-Term Human Participation Assessment in Collaborative Learning Environments Using Dynamic Scene Analysis

Audio-visual child-adult speaker classification in dyadic interactions

No-audio speaking status detection in crowded settings via visual pose-based filtering and wearable acceleration

Audio-visual voice activity detection using diffusion maps

Fine-Grained Classroom Activity Detection from Audio with Neural Networks

Multimodal behavior analysis in computer-enabled laboratories using nonverbal cues

Data Drive Development-Multimodal Measurement of Classroom Interaction

Finding Dory in the Crowd: Detecting Social Interactions using Multi-Modal Mobile Sensing

AudioInsight: Detecting Social Contexts Relevant to Social Anxiety from Speech

Toward Automated Classroom Observation: Multimodal Machine Learning to Estimate CLASS Positive Climate and Negative Climate

Social Group Detection Based on Multi-Level Consistent Behaviour Characteristics

Voice Detection and Music Smart Classroom Teaching Application Based on Mobile Edge Computing

Recurrence Quantification Analysis of Crowd Sound Dynamics

A Machine Learning Approach to Assess Student Group Collaboration Using Individual Level Behavioral Cues

Multi-Scale Audio Spectrogram Transformer for Classroom Teaching Interaction Recognition

SALSA: A Novel Dataset for Multimodal Group Behavior Analysis