Abstract:Recent research advance has significantly improved the visual realism of immersive 3D video communication. In this work we present a method to further enhance this immersive experience by adding the hand touch capability ("remote hand clapping"). In our system, each meeting participant sits in front of a large screen with haptic feedback. The local participant can reach his hand out to the screen and perform hand clapping with the remote participant as if the two participants were only separated by a virtual glass. A key challenge in emulating the remote hand touch is the realistic rendering of the participant's hand and arm as the hand touches the screen. When the hand is very close to the screen, the RGBD data required for realistic rendering is no longer available. To tackle this challenge, we present a dual representation of the user's hand. Our dual representation not only preserves the high-quality rendering usually found in recent image-based rendering systems but also allows the hand to reach to the screen. This is possible because the dual representation includes both an image-based model and a 3D geometry-based model, with the latter driven by a hand skeleton tracked by a side view camera. In addition, the dual representation provides a distance-based fusion of the image-based and 3D geometry-based models as the hand moves closer to the screen. The result is that the image-based and 3D geometry-based models mutually enhance each other, leading to realistic and seamless rendering. Our experiments demonstrate that our method provides consistent hand contact experience between remote users and improves the immersive experience of 3D video communication.

Pull and Push: Proximity-Aware User Interface for Navigating in 3D Space Using a Handheld Camera

Exploitation of Multiplayer Interaction and Development of Virtual Puppetry Storytelling Using Gesture Control and Stereoscopic Devices.

A Bare Hand Controlled Ar Map Navigation System

2D Motion Detection Bounded Hand 3D Trajectory Tracking and Gesture Recognition under Complex Background

Promoted Motion-Sensing Gestural Interaction by Adapting to Group Users' Spatial Movements

Motion-Based Perceptual User Interface

EZ-Manipulator: Designing a Mobile, Fast, and Ambiguity-Free 3D Manipulation Interface Using Smartphones

A touchable virtual screen interaction system with handheld Kinect camera

3D Intuitive Gesture Interaction via Motion Sensing

Hand's 3D movement detection with one handheld camera.

RemoteTouch: Enhancing Immersive 3D Video Communication with Hand Touch

TouchView: Mid-Air Touch on Zoomable 2D View for Distant Freehand Selection on a Virtual Reality User Interface

Tilt & Touch: Mobile Phone For 3d Interaction

An interaction system using mixed hand gestures.

A Motion-based User Interface for the Control of Virtual Humans Performing Sports.

UCam: direct manipulation using handheld camera for 3d gesture interaction.

Gaze Modulated Disambiguation Technique for Gesture Control in 3D Virtual Objects Selection

A Non-contact Interaction Method for 3D Display Devices

Back-To-Back: A Novel Approach for Real Time 3d Hand Gesture Interaction

3DTouch: A wearable 3D input device with an optical sensor and a 9-DOF inertial measurement unit

Fast 3D Hand Estimation for Mobile Interactions