Abstract:In virtual teaching scenarios, head-mounted display (HMD) interactions often employ traditional controller and UI interactions, which are not very conducive to teaching scenarios that require hand training. Existing improvements in this area have primarily focused on replacing controllers with gesture recognition. However, the exclusive use of gesture recognition may have limitations in certain scenarios, such as complex operations or multitasking environments. This study designed and tested an interaction method that combines simple gestures with voice assistance, aiming to offer a more intuitive user experience and enrich related research. A speech classification model was developed that can be activated via a fist-clenching gesture and is capable of recognising specific Chinese voice commands to initiate various UI interfaces, further controlled by pointing gestures. Virtual scenarios were constructed using Unity, with hand tracking achieved through the HTC OpenXR SDK. Within Unity, hand rendering and gesture recognition were facilitated, and interaction with the UI was made possible using the Unity XR Interaction Toolkit. The interaction method was detailed and exemplified using a teacher training simulation system, including sample code provision. Following this, an empirical test involving 20 participants was conducted, comparing the gesture-plus-voice operation to the traditional controller operation, both quantitatively and qualitatively. The data suggests that while there is no significant difference in task completion time between the two methods, the combined gesture and voice method received positive feedback in terms of user experience, indicating a promising direction for such interactive methods. Future work could involve adding more gestures and expanding the model training dataset to realize additional interactive functions, meeting diverse virtual teaching needs.

Two-Handed Interaction Techniques in Desktop VRs

Collaboration Upon Heterogeneous Platforms - From Desktop Pc To Handheld Device

Exploitation of Multiplayer Interaction and Development of Virtual Puppetry Storytelling Using Gesture Control and Stereoscopic Devices.

Two-In-One: A Design Space for Mapping Unimanual Input into Bimanual Interactions in VR for Users with Limited Movement

Hand Interfaces: Using Hands to Imitate Objects in AR/VR for Expressive Interactions

An interaction system using mixed hand gestures.

Hand-adaptive User Interface: Improved Gestural Interaction in Virtual Reality

Dual Quaternion Based Virtual Hand Interaction Modeling

RemoteTouch: Enhancing Immersive 3D Video Communication with Hand Touch

Design and Development of a Gesture-Based Interaction System for Multi-Projector Tiled Display Wall

Designing Gaze-Based Interactions for Teleoperation: Eye Stick and Eye Click

Designing and Evaluating Hand-to-Hand Gestures with Dual Commodity Wrist-Worn Devices

Elicitation and Evaluation of Hand-based Interaction Language for 3D Conceptual Design in Mixed Reality

Interactive Design With Gesture and Voice Recognition in Virtual Teaching Environments

Wearable teleoperation controller with 2-DoF robotic arm and haptic feedback for enhanced interaction in virtual reality

Enhancing hand-object interactions in virtual reality for precision manual tasks

Design and Evaluation of Bare-Hand Interaction for Precise Manipulation of Distant Objects in AR

Development and Application of Integrated Human Machine Interaction Simulation System

Bimanual Ultrasound Mid-Air Haptics for Virtual Reality Manipulation

Experimental Analysis of Freehand Multi-Object Selection Techniques in Virtual Reality Head-Mounted Displays

Asymmetric Free-Hand Interaction on a Large Display and Inspirations for Designing Natural User Interfaces