Abstract:In virtual teaching scenarios, head-mounted display (HMD) interactions often employ traditional controller and UI interactions, which are not very conducive to teaching scenarios that require hand training. Existing improvements in this area have primarily focused on replacing controllers with gesture recognition. However, the exclusive use of gesture recognition may have limitations in certain scenarios, such as complex operations or multitasking environments. This study designed and tested an interaction method that combines simple gestures with voice assistance, aiming to offer a more intuitive user experience and enrich related research. A speech classification model was developed that can be activated via a fist-clenching gesture and is capable of recognising specific Chinese voice commands to initiate various UI interfaces, further controlled by pointing gestures. Virtual scenarios were constructed using Unity, with hand tracking achieved through the HTC OpenXR SDK. Within Unity, hand rendering and gesture recognition were facilitated, and interaction with the UI was made possible using the Unity XR Interaction Toolkit. The interaction method was detailed and exemplified using a teacher training simulation system, including sample code provision. Following this, an empirical test involving 20 participants was conducted, comparing the gesture-plus-voice operation to the traditional controller operation, both quantitatively and qualitatively. The data suggests that while there is no significant difference in task completion time between the two methods, the combined gesture and voice method received positive feedback in terms of user experience, indicating a promising direction for such interactive methods. Future work could involve adding more gestures and expanding the model training dataset to realize additional interactive functions, meeting diverse virtual teaching needs.

Research On Aircraft Virtual Assembly Technology Based On Gesture Recognition

Mid-Air Gestures for Multi-Fingered Virtual Assembly System with Leap Motion.

The Virtual Assembly Technology Based on Natural Gesture

Agricultural Machinery Virtual Assembly System Using Dynamic Gesture Recognitive Interaction Based on a CNN and LSTM Network

Software Design of Virtual Training Motion Acquisition System

Exploration and Practice of Aircraft Assembly AR Intelligent Pilot Technology

Research on Gesture Recognition and Interaction of Virtual Collaborative Disassembly Training

Interactive Design With Gesture and Voice Recognition in Virtual Teaching Environments

Research on Immersive Virtual Assembly Technology

Virtual Assembling Operation Technology of Virtual Assembly Based on Virtual Hand

Gesture Recognition in Augmented Reality Assisted Assembly Training

Design and exploration of a virtual reality-based human-computer interaction system

Glove-based virtual hand grasping for virtual mechanical assembly

A Hierarchical Method of Interaction Hand Gesture for Virtual Assembly

A New Type Haptics-Based Virtual Environment System for Assembly Training of Complex Products

A Natural Bare-Hand Interaction Method With Augmented Reality for Constraint-Based Virtual Assembly

Gesture-based target acquisition in virtual and augmented reality.

P‐4.29: The Research on Virtual Reality Field Based on Gesture Recognition

Augmented Reality Assisted Assembly Training Oriented Dynamic Gesture Recognition and Prediction

Virtual Manipulation in an Immersive Virtual Environment: Simulation of Virtual Assembly

Research and Application of Multifeature Gesture Recognition in Human-Computer Interaction Based on Virtual Reality Technology