Abstract:Volumetric video (VV) recently emerges as a new form of video application providing a photorealistic immersive 3D viewing experience with 6 degree-of-freedom (DoF), which empowers many applications such as VR, AR, and Metaverse. A key problem therein is how to stream the enormous size VV through the network with limited bandwidth. Existing works mostly focused on predicting the viewport for a tiling-based adaptive VV streaming, which however only has quite a limited effect on resource saving. We argue that the content repeatability in the viewport can be further leveraged, and for the first time, propose a client-side cache-assisted strategy that aims to buffer the repeatedly appearing VV tiles in the near future so as to reduce the redundant VV content transmission. The key challenges exist in three aspects, including (1) feature extraction and mining in 6 DoF VV context, (2) accurate long-term viewing pattern estimation and (3) optimal caching scheduling with limited capacity. In this paper, we propose CaV3, an integrated cache-assisted viewport adaptive VV streaming framework to address the challenges. CaV3 employs a Long-short term Sequential prediction model (LSTSP) that achieves accurate short-term, mid-term and long-term viewing pattern prediction with a multi-modal fusion model by capturing the viewer's behavior inertia, current attention, and subjective intention. Besides, CaV3 also contains a contextual MAB-based caching adaptation algorithm (CCA) to fully utilize the viewing pattern and solve the optimal caching problem with a proved upper bound regret. Compared to existing VV datasets only containing single or co-located objects, we for the first time collect a comprehensive dataset with sufficient practical unbounded 360° scenes. The extensive evaluation of the dataset confirms the superiority of CaV3, which outperforms the SOTA algorithm by 15.6%-43% in viewport prediction and 13%-40% in system utility.

Progressive Cache Replacement for Massive Peer-to-peer WebVR Worlds

Tile-based Proactive Virtual Reality Streaming Via Online Hierarchial Learning

Web-based virtual walkthrough of panoramas

Walking Through a Very Large Virtual Environment in Real-time.

Cloud-to-end Rendering and Storage Management for Virtual Reality in Experimental Education

BALANCING FIDELITY AND PERFOR MANCE IN VIRTUAL WALKTHROUGH

Distributed Management of Virtual Reality Digital Assets for Online Experimental Education

Implementation of Progressive Texture Transmission in Distributed Virtual Environments

Intelligent Edge Caching for Metaverse Applications Based on Grover Search

: A Web-Based Distributed Virtual Walkthrough Environment

Instant Reality: Gaze-Contingent Perceptual Optimization for 3D Virtual Reality Streaming

Design and Analysis of MEC- and Proactive Caching-Based 360 Mobile VR Video Streaming

Asymmetric Circular Projection For Dynamic Virtual Reality Video Stream Switching

VRCT: A Viewport Reconstruction-Based 360° Video Caching Solution for Tile-Adaptive Streaming

User Dynamics-Aware Edge Caching and Computing for Mobile Virtual Reality

CaV3: Cache-assisted Viewport Adaptive Volumetric Video Streaming

Joint Content and Communication Resource Allocation for Privacy-Preserving Distributed Collaborative Edge Caching

Progressive Frame Patching for FoV-based Point Cloud Video Streaming

Viewport-Aware Deep Reinforcement Learning Approach for 360$^o$ Video Caching

A Real-time Virtual Reality Adaptive Streaming System