StuArt: Individualized Classroom Observation of Students with Automatic Behavior Recognition and Tracking

Huayi Zhou,Fei Jiang,Jiaxin Si,Lili Xiong,Hongtao Lu

DOI: https://doi.org/10.48550/arXiv.2211.03127

2023-03-13

Abstract:Each student matters, but it is hardly for instructors to observe all the students during the courses and provide helps to the needed ones immediately. In this paper, we present StuArt, a novel automatic system designed for the individualized classroom observation, which empowers instructors to concern the learning status of each student. StuArt can recognize five representative student behaviors (hand-raising, standing, sleeping, yawning, and smiling) that are highly related to the engagement and track their variation trends during the course. To protect the privacy of students, all the variation trends are indexed by the seat numbers without any personal identification information. Furthermore, StuArt adopts various user-friendly visualization designs to help instructors quickly understand the individual and whole learning status. Experimental results on real classroom videos have demonstrated the superiority and robustness of the embedded algorithms. We expect our system promoting the development of large-scale individualized guidance of students. More information is in <a class="link-external link-https" href="https://github.com/hnuzhy/StuArt" rel="external noopener nofollow">this https URL</a>.

Human-Computer Interaction,Computer Vision and Pattern Recognition

What problem does this paper attempt to address?

The problem that this paper attempts to solve is: in traditional classroom teaching, it is difficult for teachers to observe and help all students in real - time, especially to provide personalized guidance. Specifically, this research aims to achieve automated and continuous observation of individual student behaviors through automatic behavior recognition and tracking technologies, so as to help teachers better understand each student's learning state and provide necessary help in a timely manner.

### Main problems
1. **Limitations of traditional classroom observation**:
- Traditional classroom observation relies on manual coding, which is both costly and difficult to promote on a large scale.
- Most of the existing automatic classroom observation systems focus on the overall class situation or single - dimensional behaviors (such as emotions, lines of sight, etc.), lacking comprehensive observation of individual students.

2. **Needs for personalized classroom observation**:
- Teachers need a tool to monitor the behavior change trends of each student in real - time, so as to quickly obtain the overall class tendency and individual learning states.
- Providing personalized teaching guidance requires a detailed understanding of each student's learning participation and behavior patterns.

### Solutions
To solve the above problems, the paper proposes an automatic individualized classroom observation system named StuArt. The system uses an RGB camera to record classroom videos and realizes automatic recognition and tracking of student behaviors through the following steps:

1. **Behavior recognition**:
- Recognize five typical student behaviors: raising hands, standing up, sleeping, yawning, and smiling. These behaviors are closely related to students' learning participation.
- Use deep - learning models such as YOLOv5 for behavior detection, and design special detection strategies for different behaviors (such as yawning and smiling).

2. **Student tracking**:
- Use seat positions (rows and columns) as unique identifiers to protect students' privacy.
- Through multi - target tracking methods, ensure continuous observation and recording of each student's behavior change trends.

3. **Visualization design**:
- Design a variety of user - friendly visualization interfaces to help teachers quickly understand individual and overall learning states. These include seat maps, heat maps, link lists, and point - flow charts, etc.

### Experimental verification
The paper verifies the effectiveness and robustness of the system through experiments. The experimental results show that the StuArt system performs better than the existing automatic classroom observation systems in real - class videos, can accurately recognize and track student behaviors, and provide valuable feedback for teachers.

In conclusion, the StuArt system aims to improve the effectiveness of classroom teaching and promote the development of large - scale personalized student guidance through automation technologies and visualization tools.