Exploring the Impact of Hand Pose and Shadow on Hand-washing Action Recognition

Shengtai Ju,Amy R. Reibman

2024-06-20

Abstract:In the real world, camera-based application systems can face many challenges, including environmental factors and distribution shift. In this paper, we investigate how pose and shadow impact a classifier's performance, using the specific application of handwashing action recognition. To accomplish this, we generate synthetic data with desired variations to introduce controlled distribution shift. Using our synthetic dataset, we define a classifier's breakdown points to be where the system's performance starts to degrade sharply, and we show these are heavily impacted by pose and shadow conditions. In particular, heavier and larger shadows create earlier breakdown points. Also, it is intriguing to observe model accuracy drop to almost zero with bigger changes in pose. Moreover, we propose a simple mitigation strategy for pose-induced breakdown points by utilizing additional training data from non-canonical poses. Results show that the optimal choices of additional training poses are those with moderate deviations from the canonical poses with 50-60 degrees of rotation.

Computer Vision and Pattern Recognition,Image and Video Processing

What problem does this paper attempt to address?

The main problem that this paper attempts to solve is to explore how hand pose and shadow affect the performance of camera - based hand - washing action recognition systems. Specifically, the researchers are concerned with:

1. **The influence of hand pose**: Will different hand poses lead to a significant decline in the performance of the recognition system? If so, at which specific angles or poses is this decline most obvious?

2. **The influence of shadow**: What is the impact of shadows of different sizes, intensities, and positions on the hand - washing action recognition system? Will shadows accelerate the degradation of system performance?

3. **Distribution shift**: Will changes in hand pose and shadow cause a shift in the data distribution, resulting in a significant difference between the model's performance on the test data and its performance during training?

To answer these questions, the researchers generated a synthetic dataset to introduce a controllable distribution shift and defined the "breakdown points" of the system, that is, the points where the system performance begins to decline sharply. In this way, they were able to quantify the specific impact of hand pose and shadow on system performance.

In addition, the study also proposed a simple mitigation strategy, that is, to improve the robustness of the system by adding training data of non - standard poses. In particular, using hand poses with a rotation angle of 50 - 60 degrees as additional training data can effectively mitigate the performance degradation caused by poses.

### Summary of main contributions:
1. **Quantitatively study the impact of hand pose on the performance of the recognition system**.
2. **Evaluate the breakdown points of the hand - washing action recognition system**.
3. **In - depth analysis of the impact of shadow on the system recognition performance**.
4. **Provide insights into dataset collection and construction for future research**.

These research results are of great significance for designing more robust hand - washing action recognition systems, especially in outdoor environments such as food processing.