Abstract:With the rapid development of human–computer interaction technologies, the new media generation demands novel learning experiences with natural interaction and immersive experience. Considering that digital storytelling is a powerful pedagogical tool for young children, in this paper, we design an immersive storytelling environment that allows multiple players to use naturally interactive hand gestures to manipulate virtual puppetry for assisting narration. A set of multimodal interaction techniques is presented for a hybrid user interface that integrates existing 3D visualization and interaction devices including head‐mounted displays and depth motion sensor. In this system, the young players could intuitively use hand gestures to manipulate virtual puppets to perform a story and interact with props in a virtual stereoscopic environment. We have conducted a user experiment with four young children for pedagogical evaluation, as well as system acceptability and interactivity evaluation by postgraduate students. The results show that our framework has great potential to stimulate learning abilities of young children through collaboration tasks. The stereoscopic head‐mounted display outperformed the traditional monoscopic display in a comparison between the two.

Multi-Modal Interaction in Handheld Mobile Computing

Research on Man-Machine Interaction of Handheld Mobile Computing

A Comparative Study of Map Exploration Interfaces for Multi-Touch Tabletops.

Multi-user collaborative exploration of map and decision-making on a multi-touch tabletop

COMTIS: Customizable Touchless Interaction System for Large Screen Visualization

A Mobile Interactive System Generation Framework from Conceptual Model to Development Paradigm

A novel interactive exhibition method based on handheld devices

Exploitation of Multiplayer Interaction and Development of Virtual Puppetry Storytelling Using Gesture Control and Stereoscopic Devices.

Multimodal User Interfaces for a Travel Assistant

An interaction system using mixed hand gestures.

An Agent-Based Multimodal User Interfaces Middleware for Ubiquitous Computing Environments: Intelligent Exhibition Hall

Mobile assistant: enhancing desktop interaction using mobile phone

Gesture-based Visualization Interaction Design in Handheld Devices

A Review of Multimodal Interaction

Review of Sensory Feedback Simulation Methods in Multi-modal Human-computer Interaction

3D Intuitive Gesture Interaction via Motion Sensing

TGH: a case study of designing natural interaction for mobile guide systems

A Multimodal Home Entertainment Interface Via a Mobile Device

Multimodal interaction: A review

SimUser: Generating Usability Feedback by Simulating Various Users Interacting with Mobile Applications

Study on Human-Computer Interaction Framework of Pervasive Computing