Abstract:Recent research advance has significantly improved the visual realism of immersive 3D video communication. In this work we present a method to further enhance this immersive experience by adding the hand touch capability ("remote hand clapping"). In our system, each meeting participant sits in front of a large screen with haptic feedback. The local participant can reach his hand out to the screen and perform hand clapping with the remote participant as if the two participants were only separated by a virtual glass. A key challenge in emulating the remote hand touch is the realistic rendering of the participant's hand and arm as the hand touches the screen. When the hand is very close to the screen, the RGBD data required for realistic rendering is no longer available. To tackle this challenge, we present a dual representation of the user's hand. Our dual representation not only preserves the high-quality rendering usually found in recent image-based rendering systems but also allows the hand to reach to the screen. This is possible because the dual representation includes both an image-based model and a 3D geometry-based model, with the latter driven by a hand skeleton tracked by a side view camera. In addition, the dual representation provides a distance-based fusion of the image-based and 3D geometry-based models as the hand moves closer to the screen. The result is that the image-based and 3D geometry-based models mutually enhance each other, leading to realistic and seamless rendering. Our experiments demonstrate that our method provides consistent hand contact experience between remote users and improves the immersive experience of 3D video communication.

Real-time whiteboard capture and processing using a video camera for teleconferencing

Video Conference System for Enhancing Quality of Target Region under Low Bit Rate

Stereo Camera System and Its Application in Networked Virtual Environment

An Automated End-to-end Lecture Capture and Broadcasting System

A portable solution for automatic lecture room camera management

Wide field of view and two-dimensional position measurement system based on linear CCD

Join the Workshop Better with Telepresence Capturer.

SRec: An Automatic Slide Capturing and Sharing System

Building an Intelligent Camera Management System

Tele-Aloha: A Low-budget and High-authenticity Telepresence System Using Sparse RGB Cameras

Automating camera management for lecture room environments.

A Streaming-Based Approach for Remote Interaction of the Multi-Channel Display System for Group Users

Tele-Aloha: A Telepresence System with Low-budget and High-authenticity Using Sparse RGB Cameras

V-Hands: Touchscreen-based Hand Tracking for Remote Whiteboard Interaction

RemoteTouch: Enhancing Immersive 3D Video Communication with Hand Touch

Semi-Videoconference System Using Real-Time Wireless Technologies.

Distributed meetings: a meeting capture and broadcasting system.

42.4L: Late‐News Paper: Footprint of Scalable 3D Telecommunication: Using Integral Light Field Display and Kinect‐based Capture

An Internet-Based Conference System for Real-Time Distributed Design Evaluation

Bringing Telepresence to Every Desk

Toward A Better User Experience in Tele-education - Recent Advance in Smart Classroom Project