Abstract:In recent years, workplaces and educational institutes have widely adopted virtual meeting platforms. This has led to a growing interest in analyzing and extracting insights from these meetings, which requires effective detection and tracking of unique individuals. In practice, there is no standardization in video meetings recording layout, and how they are captured across the different platforms and services. This, in turn, creates a challenge in acquiring this data stream and analyzing it in a uniform fashion. Our approach provides a solution to the most general form of video recording, usually consisting of a grid of participants (\cref{fig:videomeeting}) from a single video source with no metadata on participant locations, while using the least amount of constraints and assumptions as to how the data was acquired. Conventional approaches often use YOLO models coupled with tracking algorithms, assuming linear motion trajectories akin to that observed in CCTV footage. However, such assumptions fall short in virtual meetings, where participant video feed window can abruptly change location across the grid. In an organic video meeting setting, participants frequently join and leave, leading to sudden, non-linear movements on the video grid. This disrupts optical flow-based tracking methods that depend on linear motion. Consequently, standard object detection and tracking methods might mistakenly assign multiple participants to the same tracker. In this paper, we introduce a novel approach to track and re-identify participants in remote video meetings, by utilizing the spatio-temporal priors arising from the data in our domain. This, in turn, increases tracking capabilities compared to the use of general object tracking. Our approach reduces the error rate by 95% on average compared to YOLO-based tracking methods as a baseline.
What problem does this paper attempt to address?
This paper attempts to address the challenges of tracking and re - identifying individuals in multi - participant virtual meetings. Specifically, the paper focuses on the following issues:
1. **Non - standardization of video layout**: Different platforms and services capture and record video conferences in different ways, leading to inconsistencies in data stream acquisition and analysis.
2. **Dynamic changes of participants**: Participants frequently join, leave the meeting, turn on or off their cameras, and change devices during the meeting (such as switching from a personal computer to a mobile device). These behaviors render traditional tracking methods based on linear motion trajectories ineffective.
3. **Non - linear motion in the video grid**: The position of a participant's video window in the grid may change suddenly, which breaks tracking methods that rely on continuity and linear motion, such as optical flow.
4. **Limitations of existing methods**: Traditional object detection and tracking methods (such as the YOLO model combined with tracking algorithms) assume that the object motion trajectory is linear, similar to the situation in CCTV surveillance. However, in virtual meetings, this assumption usually does not hold, leading to an increase in tracking errors.
To solve these problems, the paper proposes a new method that utilizes spatio - temporal prior information in video conference data to track and re - identify participants. This method reduces the tracking error rate by approximately 95% and is a significant improvement compared to YOLO - based tracking methods. By effectively managing the dynamic characteristics in virtual meetings, this method ensures reliable participant identification, maintaining high accuracy even in complex scenarios with fluctuating numbers of participants and diverse device usage.
### Core contributions of the paper
- **Innovative tracking algorithm**: Proposed an algorithm that can accurately track and re - identify participants in a dynamic virtual meeting environment.
- **Application of spatio - temporal prior information**: Utilized the spatio - temporal characteristics unique to video conferences to improve the accuracy and robustness of tracking.
- **Reduction of tracking errors**: Compared to existing YOLO - based tracking methods, the new method reduces the error rate by approximately 95%.
### Formula summary
The key formulas involved in the paper include:
- The set of video frames \(F=\{f_0, f_1,\ldots, f_T\}\), where \(T\) is the total number of frames.
- The set of participant tracks \(O = \{O_1, O_2,\ldots, O_k\}\), where \(k\) is the actual number of participants.
- Each track \(O_i=\{o_{ft}, o_{ft + 1},\ldots, o_{ft+\Delta t}\}\), representing the detection results of participant \(i\) at different time points \(t\).
- The facial embedding sequence \(E_i=\text{FaceNet}(O_i)\), which is used to extract the unique features of participants.
Through these formulas and methods, the paper provides an effective solution to the complex challenges of participant tracking in virtual meetings.