Abstract:In virtual teaching scenarios, head-mounted display (HMD) interactions often employ traditional controller and UI interactions, which are not very conducive to teaching scenarios that require hand training. Existing improvements in this area have primarily focused on replacing controllers with gesture recognition. However, the exclusive use of gesture recognition may have limitations in certain scenarios, such as complex operations or multitasking environments. This study designed and tested an interaction method that combines simple gestures with voice assistance, aiming to offer a more intuitive user experience and enrich related research. A speech classification model was developed that can be activated via a fist-clenching gesture and is capable of recognising specific Chinese voice commands to initiate various UI interfaces, further controlled by pointing gestures. Virtual scenarios were constructed using Unity, with hand tracking achieved through the HTC OpenXR SDK. Within Unity, hand rendering and gesture recognition were facilitated, and interaction with the UI was made possible using the Unity XR Interaction Toolkit. The interaction method was detailed and exemplified using a teacher training simulation system, including sample code provision. Following this, an empirical test involving 20 participants was conducted, comparing the gesture-plus-voice operation to the traditional controller operation, both quantitatively and qualitatively. The data suggests that while there is no significant difference in task completion time between the two methods, the combined gesture and voice method received positive feedback in terms of user experience, indicating a promising direction for such interactive methods. Future work could involve adding more gestures and expanding the model training dataset to realize additional interactive functions, meeting diverse virtual teaching needs.

Commanding and Re-Dictation

Toward Interactive Dictation

Just Speak It: Minimize Cognitive Load for Eyes-Free Text Editing with a Smart Voice Assistant

Rewriting the Script: Adapting Text Instructions for Voice Interaction

Improving Error Correction and Text Editing Using Voice and Mouse Multimodal Interface

Typist Experiment: an Investigation of Human-to-Human Dictation via Role-play to Inform Voice-based Text Authoring

AutoTask: Executing Arbitrary Voice Commands by Exploring and Learning from Mobile GUI

Automatically Generating and Improving Voice Command Interface from Operation Sequences on Smartphones

Cross-modal Task Understanding and Execution of Voice-fingertip Reading Instruction by Using Small Family Service Robotic

EasyCmd: Navigation by Voice Commands

Voicify Your UI: Towards Android App Control with Voice Commands

EditScribe: Non-Visual Image Editing with Natural Language Verification Loops

EyeSayCorrect: Eye Gaze and Voice Based Hands-free Text Correction for Mobile Devices

Scenario-Based Programming of Voice-Controlled Medical Robotic Systems

Interactive Design With Gesture and Voice Recognition in Virtual Teaching Environments

Lip-Interact: Improving Mobile Device Interaction with Silent Speech Commands.

A new conversational interaction concept for document creation and editing on mobile devices for visually impaired users

Listen, Chat, and Edit: Text-Guided Soundscape Modification for Enhanced Auditory Experience

Assessing User Needs in Non-Visual Text Input: Perceptions of Blind Adults on Current and Experimental Mobile Interfaces

Voice Attribute Editing with Text Prompt

Voice Assistant for Blind Person