Abstract:Recent research advance has significantly improved the visual realism of immersive 3D video communication. In this work we present a method to further enhance this immersive experience by adding the hand touch capability ("remote hand clapping"). In our system, each meeting participant sits in front of a large screen with haptic feedback. The local participant can reach his hand out to the screen and perform hand clapping with the remote participant as if the two participants were only separated by a virtual glass. A key challenge in emulating the remote hand touch is the realistic rendering of the participant's hand and arm as the hand touches the screen. When the hand is very close to the screen, the RGBD data required for realistic rendering is no longer available. To tackle this challenge, we present a dual representation of the user's hand. Our dual representation not only preserves the high-quality rendering usually found in recent image-based rendering systems but also allows the hand to reach to the screen. This is possible because the dual representation includes both an image-based model and a 3D geometry-based model, with the latter driven by a hand skeleton tracked by a side view camera. In addition, the dual representation provides a distance-based fusion of the image-based and 3D geometry-based models as the hand moves closer to the screen. The result is that the image-based and 3D geometry-based models mutually enhance each other, leading to realistic and seamless rendering. Our experiments demonstrate that our method provides consistent hand contact experience between remote users and improves the immersive experience of 3D video communication.

Magic Props: A Multi-sensory System Fusing Virtual Effects in Live Drama Performance Spatially

Exploitation of Multiplayer Interaction and Development of Virtual Puppetry Storytelling Using Gesture Control and Stereoscopic Devices.

Exploitation Of Novel Multiplayer Gesture-Based Interaction And Virtual Puppetry For Digital Storytelling To Develop Children'S Narrative Skills

Restoration of traditional Chinese shadow play-Piying art from tangible interaction

Research and realization of stage augmented reality system

Application of Physical Interactive Mixed Reality System Based on MLAT in the Field of Stage Performance

Magic music desk: a multi-modal embodied interactive desk

Magic Music Desk: A Tangible and Ubiquitous Multimodal Music and Speech System

Real-Time Motion Capture Technology on a Live Theatrical Performance with Computer Generated Scenery

A touchable virtual screen interaction system with handheld Kinect camera

A Penetrable Interactive 3D Display Based on Motion Recognition （invited Paper）

P‐78: an Interactive 360° Floating 3D Display Based on Gesture Recognition

Real-time Digitised Shadow Play Performance Method Based on Multi-Point Interactive Controlling Method

RemoteTouch: Enhancing Immersive 3D Video Communication with Hand Touch

A Real-Time Interactive System of Surface Reconstruction and Dynamic Projection Mapping with RGB-depth Sensor and Projector.

42.4L: Late‐News Paper: Footprint of Scalable 3D Telecommunication: Using Integral Light Field Display and Kinect‐based Capture

Live Mixed Reality Video Production for Educational Presentation

Magic Land : a 3 D Human Capture Mixed Reality System for Museum Experiences

MagicChem: a MR system based on needs theory for chemical experiments

Design Mode of Stage Performing Arts Based on 3D Modeling and Moving Edge Computing Technology

Demo: Creating Interactive Virtual Zones in Physical Space with Magnetic-Induction