Abstract:With the rapid development of human–computer interaction technologies, the new media generation demands novel learning experiences with natural interaction and immersive experience. Considering that digital storytelling is a powerful pedagogical tool for young children, in this paper, we design an immersive storytelling environment that allows multiple players to use naturally interactive hand gestures to manipulate virtual puppetry for assisting narration. A set of multimodal interaction techniques is presented for a hybrid user interface that integrates existing 3D visualization and interaction devices including head‐mounted displays and depth motion sensor. In this system, the young players could intuitively use hand gestures to manipulate virtual puppets to perform a story and interact with props in a virtual stereoscopic environment. We have conducted a user experiment with four young children for pedagogical evaluation, as well as system acceptability and interactivity evaluation by postgraduate students. The results show that our framework has great potential to stimulate learning abilities of young children through collaboration tasks. The stereoscopic head‐mounted display outperformed the traditional monoscopic display in a comparison between the two.

Magic Music Desk: A Tangible and Ubiquitous Multimodal Music and Speech System

Magic music desk: a multi-modal embodied interactive desk

Multisensory Musical Entertainment Systems

MagicWatch: interacting & segueing.

Exploitation of Multiplayer Interaction and Development of Virtual Puppetry Storytelling Using Gesture Control and Stereoscopic Devices.

Mind Band: A Crossmedia AI Music Composing Platform

MDS: a multimodal-based dialog system

VisibleSound: Perceiving Environmental Sound with 4D Form

Motion-Based Perceptual User Interface

Musinger: Communication of Music over a Distance with Wearable Haptic Display and Touch Sensitive Surface

Tangible Mixed Reality Desktop for Digital Media Management

Interaction With Gaze, Gesture, and Speech in a Flexibly Configurable Augmented Reality System

Game-City: A Ubiquitous Large Area Multi-Interface Mixed Reality Game Space for Wearable Computers

A New Mmwave-Speech Multimodal Speech System for Voice User Interface

A Multimodal Home Entertainment Interface Via a Mobile Device

Application of the Multimodal Human-Computer Interaction Technology in Product Virtual Display

A gesture- and head-based multimodal interaction platform for MR remote collaboration

Designing, Playing, and Performing with a Vision-based Mouth Interface

H.: Magic Land: a 3D Human Capture Mixed Reality System for Museum Experiences

Designing User Interaction using Gesture and Speech for Mixed Reality Interface

Dream-Experiment: A MR User Interface with Natural Multi-channel Interaction for Virtual Experiments