CaSAR: Contact-aware Skeletal Action Recognition

Junan Lin,Zhichao Sun,Enjie Cao,Taein Kwon,Mahdi Rad,Marc Pollefeys

2023-09-17

Abstract:Skeletal Action recognition from an egocentric view is important for applications such as interfaces in AR/VR glasses and human-robot interaction, where the device has limited resources. Most of the existing skeletal action recognition approaches use 3D coordinates of hand joints and 8-corner rectangular bounding boxes of objects as inputs, but they do not capture how the hands and objects interact with each other within the spatial context. In this paper, we present a new framework called Contact-aware Skeletal Action Recognition (CaSAR). It uses novel representations of hand-object interaction that encompass spatial information: 1) contact points where the hand joints meet the objects, 2) distant points where the hand joints are far away from the object and nearly not involved in the current action. Our framework is able to learn how the hands touch or stay away from the objects for each frame of the action sequence, and use this information to predict the action class. We demonstrate that our approach achieves the state-of-the-art accuracy of 91.3% and 98.4% on two public datasets, H2O and FPHA, respectively.

Computer Vision and Pattern Recognition

What problem does this paper attempt to address?

### Problems Addressed by the Paper

The paper aims to address the problem of skeletal action recognition from a first-person perspective, particularly in applications such as Augmented Reality (AR)/Virtual Reality (VR) glasses and human-computer interaction. Most current skeletal action recognition methods primarily rely on the 3D coordinates of hand joints and the 8-corner bounding box of objects, but these methods overlook the spatial interaction information between the hand and the object. Therefore, this paper proposes a new framework—Contact-aware Skeletal Action Recognition (CaSAR), which introduces a new representation method for hand-object interaction to address this shortcoming.

Specifically, the CaSAR framework includes the following innovations:
1. **Contact Points and Distant Points**: Capturing the spatial relationship between the hand and the object through contact points (positions where hand joints directly touch the object) and distant points (positions where hand joints are away from the object).
2. **Contact Map**: Using a contact map to represent hand-object interaction information, thereby better understanding the spatial relationship between the hand and the object.
3. **No Need for 3D Mesh Models**: Unlike previous methods based on 3D meshes, CaSAR only requires hand joint positions and the 8-corner bounding box of objects for training, without the need for additional 3D mesh models.

Through the above methods, CaSAR achieves state-of-the-art accuracy on two public datasets, H2O and FPHA, with 91.3% and 98.4% respectively. This indicates that the method can significantly improve the accuracy of action recognition from a first-person perspective and has broad practical application potential.