Abstract:Recent research advance has significantly improved the visual realism of immersive 3D video communication. In this work we present a method to further enhance this immersive experience by adding the hand touch capability ("remote hand clapping"). In our system, each meeting participant sits in front of a large screen with haptic feedback. The local participant can reach his hand out to the screen and perform hand clapping with the remote participant as if the two participants were only separated by a virtual glass. A key challenge in emulating the remote hand touch is the realistic rendering of the participant's hand and arm as the hand touches the screen. When the hand is very close to the screen, the RGBD data required for realistic rendering is no longer available. To tackle this challenge, we present a dual representation of the user's hand. Our dual representation not only preserves the high-quality rendering usually found in recent image-based rendering systems but also allows the hand to reach to the screen. This is possible because the dual representation includes both an image-based model and a 3D geometry-based model, with the latter driven by a hand skeleton tracked by a side view camera. In addition, the dual representation provides a distance-based fusion of the image-based and 3D geometry-based models as the hand moves closer to the screen. The result is that the image-based and 3D geometry-based models mutually enhance each other, leading to realistic and seamless rendering. Our experiments demonstrate that our method provides consistent hand contact experience between remote users and improves the immersive experience of 3D video communication.

A Timestamp-Independent Haptic-Visual Synchronization Method for Haptic-Based Interaction System

A Vibro-tactile Stimulation and Vibro-signature Synchronization Device for SSSEP-Based Study.

Visual Attention and Haptic Control: A Cross-Study.

Suspended Three-Dimensional Display Haptic Interaction Based on Leap Motion

A Lightweight Haptic Interface for Hand-to-Object Tasks with Spatiotemporal Displays

Development of an Integrated Haptic Sensor System for Multimodal Human–Computer Interaction Using Ultrasonic Array and Cable Robot

Synchronization-Oriented Placement and Retrieval Strategies for Delay-Sensitive Media Streams

Development of a Synchronized Human-Robot-Virtuality Interaction System Using Cooperative Robot and Motion Capture Device

Haptic interface with flexible self-sensing actuators for wireless touch communication

Content-Aware Cross-Modal Stream Transmission

Heterogeneous Stream Scheduling for Cross-Modal Transmission

RemoteTouch: Enhancing Immersive 3D Video Communication with Hand Touch

Cross-Modal Collaborative Communications

Investigating the Effect of Sensory Concurrency on Learning Haptic Spatiotemporal Signals

Multi-Object Multimedia Presentation Synchronization Strategy

Encoding of tactile information in hand via skin-integrated wireless haptic interface

A human-in-the-loop haptic interaction with subjective evaluation

Tactile Codec with Visual Assistance in Multi-modal Communication for Digital Health

Design and Research of Multimodal Fusion Feedback Device Based on Virtual Interactive System

Video2Haptics: Converting Video Motion to Dynamic Haptic Feedback with Bio-Inspired Event Processing

Toward Spatial Temporal Consistency of Joint Visual Tactile Perception in VR Applications