Abstract:Body actions and head gestures are natural interfaces for interaction in virtual environments. Existing methods for in-place body action recognition often require hardware more than a head-mounted display (HMD), making body action interfaces difficult to be introduced to ordinary virtual reality (VR) users as they usually only possess an HMD. In addition, there lacks a unified solution to recognize in-place body actions and head gestures. This potentially hinders the exploration of the use of in-place body actions and head gestures for novel interaction experiences in virtual environments. We present a unified two-stream 1-D convolutional neural network (CNN) for recognition of body actions when a user performs walking-in-place (WIP) and for recognition of head gestures when a user stands still wearing only an HMD. Compared to previous approaches, our method does not require specialized hardware and/or additional tracking devices other than an HMD and can recognize a significantly larger number of body actions and head gestures than other existing methods. In total, ten in-place body actions and eight head gestures can be recognized with the proposed method, which makes this method a readily available body action interface (head gestures included) for interaction with virtual environments. We demonstrate one utility of the interface through a virtual locomotion task. Results show that the present body action interface is reliable in detecting body actions for the VR locomotion task but is physically demanding compared to a touch controller interface. The present body action interface is promising for new VR experiences and applications, especially for VR fitness applications where workouts are intended.

Real-time Egocentric Gesture Recognition on Mobile Head Mounted Displays

Dynamic hand gesture recognition using hidden Markov models

2D Motion Detection Bounded Hand 3D Trajectory Tracking and Gesture Recognition under Complex Background

RGC: Reliable Gesture Classification Via Wearables Using GANs-Based Data Augmentation.

A real-time hand gesture recognition method

Egocentric Gesture Recognition for Head-Mounted AR devices

Real-Time Hand Gesture Recognition Using RGB-D Sensor

Gaze Gestures and Their Applications in human-computer interaction with a head-mounted display

Egocentric 6-DoF Tracking of Small Handheld Objects

Real Time 3D-Handwritten Character and Gesture Recognition for Smartphone

Facial Expressions Recognition Based on Convolutional Neural Networks for Mobile Virtual Reality.

Real-Time Gesture Recognition with Virtual Glove Markers

Real-Time Recognition of In-Place Body Actions and Head Gestures using Only a Head-Mounted Display

A Real Time Vision-Based Hand Gestures Recognition System

Head and Eye Egocentric Gesture Recognition for Human-Robot Interaction Using Eyewear Cameras

Design and evaluation of a hand gesture recognition approach for real-time interactions

Event-Based Gesture Recognition With Dynamic Background Suppression Using Smartphone Computational Capabilities

x-RAGE: eXtended Reality -- Action & Gesture Events Dataset

Real World Hand Gesture Interaction in Virtual Reality

Synthetic Video Generation for Robust Hand Gesture Recognition in Augmented Reality Applications

A Novel Autonomous Learning Framework to Enhance Semg-Based Hand Gesture Recognition Using Depth Information.