Abstract:Point cloud video (PCV) offers watching experiences in photorealistic 3D scenes with six-degree-of-freedom (6-DoF), enabling a variety of VR and AR applications. The user's Field of View (FoV) is more fickle with 6-DoF movement than 3-DoF movement in 360-degree video. PCV streaming is extremely bandwidth-intensive. However, current streaming systems require hundreds of Mbps bandwidth, exceeding the bandwidth capabilities of commodity devices. To save bandwidth, FoV-adaptive streaming predicts a user's FoV and only downloads point cloud data falling in the predicted FoV. But it is difficult to accurately predict the user's FoV even 2-3 seconds before playback due to 6-DoF. Misprediction of FoV or network bandwidth dips results in frequent stalls. To avoid rebuffering, existing systems would cause incomplete FoV and degraded experience, deteriorating the user's quality of experience (QoE). In this paper, we describe Fumos, a novel system that preserves interactive experience by avoiding playback stalls while maintaining high perceptual quality and high compression rate. We find a research gap in inter-frame redundant utilization and progressive mechaism. Fumos has three crucial designs, including (1) Neural compression framework with inter-frame coding, namely N-PCC, which achieves both bandwidth efficiency and high fidelity. (2) Progressive refinement streaming framework that enables continuous playback by incrementally upgrading a fetched portion to a higher quality (3) System-level adaptation that employs Lyapunov optimization to jointly optimize the long-term user QoE. Experimental results demonstrate that Fumos significantly outperforms Draco, achieving an average decoding rate acceleration of over 260×. Moreover, the proposed compression framework N-PCC attains remarkable BD-Rate gains, averaging 91.7% and 51.7% against the state-of-the-art point cloud compression methods G-PCC and V-PCC, respectively.

Point Cloud Video Streaming: Challenges and Solutions

Towards Holographic Video Communications: A Promising AI-driven Solution

Demo Abstract: Narwhal: a DASH-based Point Cloud Video Streaming System over Wireless Networks

Fumos: Neural Compression and Progressive Refinement for Continuous Point Cloud Video Streaming

Low-bitrate Volumetric Video Streaming with Depth Image

Design and Analysis of MEC- and Proactive Caching-Based 360 Mobile VR Video Streaming

Cloud Rendering-based Volumetric Video Streaming System for Mixed Reality Services

From Capture to Display: A Survey on Volumetric Video

Optimal Volumetric Video Streaming with Hybrid Saliency based Tiling

Instant Reality: Gaze-Contingent Perceptual Optimization for 3D Virtual Reality Streaming

CUBIST: High-Quality 360-Degree Video Streaming Services via Tile-based Edge Caching and FoV-Adaptive Prefetching

A QoE Model in Point Cloud Video Streaming

LiveVV: Human-Centered Live Volumetric Video Streaming System

Bits-to-Photon: End-to-End Learned Scalable Point Cloud Compression for Direct Rendering

Towards Optimal Real-time Volumetric Video Streaming: A Rolling Optimization and Deep Reinforcement Learning Based Approach

Progressive Frame Patching for FoV-based Point Cloud Video Streaming

Panoramic Video Streaming Based on Splitting

Optimal Wireless Streaming of Multi-Quality 360 VR Video By Exploiting Natural, Relative Smoothness-Enabled, and Transcoding-Enabled Multicast Opportunities

Virtual Reality Telepresence: 360-Degree Video Streaming with Edge-Compute Assisted Static Foveated Compression

Dynamic 3D Point Cloud Streaming: Distortion and Concealment

Spatial Perceptual Quality Aware Adaptive Volumetric Video Streaming