Tele-Aloha: A Telepresence System with Low-budget and High-authenticity Using Sparse RGB Cameras
Hanzhang Tu,Ruizhi Shao,Xue Dong,Shunyuan Zheng,Hao Zhang,Lili Chen,Meili Wang,Wenyu Li,Siyan Ma,Shengping Zhang,Boyao Zhou,Yebin Liu
DOI: https://doi.org/10.1145/3641519.3657491
2024-01-01
Abstract:In this paper, we present a low-budget and high-authenticity bidirectionaltelepresence system, Tele-Aloha, targeting peer-to-peer communicationscenarios. Compared to previous systems, Tele-Aloha utilizes only four sparseRGB cameras, one consumer-grade GPU, and one autostereoscopic screen to achievehigh-resolution (2048x2048), real-time (30 fps), low-latency (less than 150ms)and robust distant communication. As the core of Tele-Aloha, we propose anefficient novel view synthesis algorithm for upper-body. Firstly, we design acascaded disparity estimator for obtaining a robust geometry cue. Additionallya neural rasterizer via Gaussian Splatting is introduced to project latentfeatures onto target view and to decode them into a reduced resolution.Further, given the high-quality captured data, we leverage weighted blendingmechanism to refine the decoded image into the final resolution of 2K.Exploiting world-leading autostereoscopic display and low-latency iristracking, users are able to experience a strong three-dimensional sense evenwithout any wearable head-mounted display device. Altogether, our telepresencesystem demonstrates the sense of co-presence in real-life experiments,inspiring the next generation of communication.