Evaluating Gesture Recognition in Virtual Reality

Sandeep Reddy Sabbella,Sara Kaszuba,Francesco Leotta,Pascal Serrarens,Daniele Nardi

2024-01-09

Abstract:Human-Robot Interaction (HRI) has become increasingly important as robots are being integrated into various aspects of daily life. One key aspect of HRI is gesture recognition, which allows robots to interpret and respond to human gestures in real-time. Gesture recognition plays an important role in non-verbal communication in HRI. To this aim, there is ongoing research on how such non-verbal communication can strengthen verbal communication and improve the system's overall efficiency, thereby enhancing the user experience with the robot. However, several challenges need to be addressed in gesture recognition systems, which include data generation, transferability, scalability, generalizability, standardization, and lack of benchmarking of the gestural systems. In this preliminary paper, we want to address the challenges of data generation using virtual reality simulations and standardization issues by presenting gestures to some commands that can be used as a standard in ground robots.

Human-Computer Interaction,Robotics

What problem does this paper attempt to address?

This paper explores the application of virtual reality (VR) in solving gesture recognition problems, particularly in the field of human-robot interaction (HRI). Gesture recognition is a key component of HRI that enables robots to understand and respond to nonverbal signals from humans. Although there are existing datasets for sign language recognition, generating practical and feasible gesture data for HRI is still scarce.

The paper identifies data generation, transferability, scalability, generalizability, standardization, and lack of benchmark tests as the main challenges faced by gesture recognition systems. The researchers propose using VR simulations to generate gesture datasets and address the issue of data generation in this way. They also focus on the standardization problem and propose a set of standardized gesture commands that can be used for ground robots.

The related work section mentions the successful application of deep learning methods, such as convolutional neural networks and recurrent neural networks, in real-time gesture recognition. The researchers also discuss the progress in recognition, generation, and animation techniques for hand and body movements in VR.

The paper describes their approach, which includes defining gestures, collecting real-world data using Intel Realsense Depth cameras, and generating additional data through VR simulations. They use virtual humanoid avatars to perform gestures in virtual environments, creating a large number of annotated data samples.

The experimental section is planned to be divided into three stages, evaluating the performance of models trained on mixed data (virtual and real data) in real-world environments. The researchers hope to answer questions about the optimal data ratio, cross-environment applicability of a single model, performance of the model on real data, and effective evaluation metrics through these experiments.

In the conclusion, the authors emphasize the advantages of VR simulations in acquiring real-world data but also highlight limitations such as cost and complexity of setup. They plan to validate the impact of VR data on improving recognition system performance by comparing the network performance with different training data and ultimately applying this approach to practical robots.