Abstract:Recent research advance has significantly improved the visual realism of immersive 3D video communication. In this work we present a method to further enhance this immersive experience by adding the hand touch capability ("remote hand clapping"). In our system, each meeting participant sits in front of a large screen with haptic feedback. The local participant can reach his hand out to the screen and perform hand clapping with the remote participant as if the two participants were only separated by a virtual glass. A key challenge in emulating the remote hand touch is the realistic rendering of the participant's hand and arm as the hand touches the screen. When the hand is very close to the screen, the RGBD data required for realistic rendering is no longer available. To tackle this challenge, we present a dual representation of the user's hand. Our dual representation not only preserves the high-quality rendering usually found in recent image-based rendering systems but also allows the hand to reach to the screen. This is possible because the dual representation includes both an image-based model and a 3D geometry-based model, with the latter driven by a hand skeleton tracked by a side view camera. In addition, the dual representation provides a distance-based fusion of the image-based and 3D geometry-based models as the hand moves closer to the screen. The result is that the image-based and 3D geometry-based models mutually enhance each other, leading to realistic and seamless rendering. Our experiments demonstrate that our method provides consistent hand contact experience between remote users and improves the immersive experience of 3D video communication.

Immersive Virtual Tour Using Multi-projection with Remote Cluster Rendering

Immersive Display and Interactive Techniques for the Modeling and Rendering of Virtual Heritage Environments

Modeling and immersive display of cultural heritage

Web-based virtual walkthrough of panoramas

A Remote Interactive Exhibition Method

Dunhuang Cave285 multimedia integrated virtual exhibit

Image-Based Virtual Navigation System for Art Caves.

Dunhuang Art Cave - Presentation And Preservation Based On Ve

Picture Based Virtual Touring

Virtual Real-Time Walkthrough of Multi-Spectral Scene

Building Virtual Entertainment Environment with Tiled Display Wall and Motion Tracking.

Building High Performance DVR Via HLA, Scene Graph and Parallel Rendering.

RemoteTouch: Enhancing Immersive 3D Video Communication with Hand Touch

Virtual Walk-Through of Dunhuang Panorama-Based on Web

Real time stereo rendering for augmented reality on 3DTV system

Multi-Projector Tiled Display Wall Calibration With A Camera

An Efficient Image-Based Virtual Tour System

A Real-Time Interactive System of Surface Reconstruction and Dynamic Projection Mapping with RGB-depth Sensor and Projector.

HARNESSING SMARTPHONE USERS 'CONTRIBUTION FOR VIRTUAL TOUR CONSTRUCTION

Interactive Browsing of Large Images on Multi-Projector Display Wall System

360-degree large-scale multiprojection light-field 3D display system.