Abstract:Immersive video applications grow faster for users to freely navigate within a virtualized 3D environment for entertainment, productivity, training, etc. Fundamentally, such system can be facilitated by an interactive Gigapixel Video Streaming (iGVS) platform from array camera capturing to end user interaction. This interactive system demands a large amount of network bandwidth to sustain the reliable service provisioning, hindering its massive market adoption. Thus, we propose to segment the gigapixel scene into non-overlapped spatial tiles. Each tile only covers a sub-region of the entire scene. One or more tiles will be used to represent an instantaneous viewport interested by a specific user. Tiles are then encoded at a variety of quality scales using various combinations of spatial, temporal and amplitude resolutions (STAR), which are typically encapsulated into temporally-aligned tile video chunks (or simply chunks). Chunks at different quality level can be processed in parallel for real-time purpose. With such setup, diverse chunk combinations can be simultaneously accessed by heterogeneous user per its request, and viewport-adaptation based content navigation in an immersive space can be also realized by adapting multiscale chunks properly, under the bandwidth constraints. A serial computational vision models measuring the perceptual quality of viewport video in terms of its quality scales, adaptation factors, as well as the peripheral vision thresholds, are devised to prepare and guide the chunk adaptation for the best perceptual quality index. Furthermore, in response to the time-varying network, a deep reinforcement learning (DRL) based adaptive real-time streaming (ARS) scheme is developed, by learning the future decision from the historical network states, to maximize the overall quality of experience (QoE) in a practical Internet-based streaming scenario. Our experiments have revealed that averaged QoE can be improved by about 60%, and its standard deviation can be also reduced by approximate to 30%, in comparison to the popular Google congestion control algorithm widely adopted in existing system for adaptive streaming, demonstrating the efficiency of our multiscale accelerated iGVS for immersive video application.

An Optimal Spatial-Temporal Smoothness Approach for Tile-Based 360-Degree Video Streaming

MTC360: A Multi-Tiles Configuration for Viewport-Dependent 360-Degree Video Streaming

Optimizing 360-Degree Video Streaming: A Hybrid Tile-Based Dynamic Adaptive Streaming Approach

A Novel Multi-scale Spatial Tiling Based Adaptive Streaming Method for 360° Video

Efficient Tile-Based Adaptive Streaming for 360-Degree Video-on-Demand Systems

SAD360: Spherical Viewport-Aware Dynamic Tiling for 360-Degree Video Streaming

A 360° Video Adaptive Streaming Scheme Based on Multiple Video Qualities.

Optimal Tile-Based Encoding for 360-Degree Video Streaming

Synergistic Temporal-Spatial User-Aware Viewport Prediction for Optimal Adaptive 360-Degree Video Streaming

TVG-Streaming : Learning User Behaviors for QoE-Optimized 360-Degree Video Streaming

TILE-BASED QOE-DRIVEN HTTP/2 STREAMING SYSTEM FOR 360 VIDEO

Tile-based Multi-source Adaptive Streaming for 360-degree Ultra-High-Definition Videos.

Exploiting Layer and Spatial Correlations to Enhance SVC and Tile Based 360-Degree Video Streaming.

Optimal Wireless Streaming of Multi-Quality 360 VR Video By Exploiting Natural, Relative Smoothness-Enabled, and Transcoding-Enabled Multicast Opportunities

Adaptive Streaming of 360 Videos With Perfect, Imperfect, and Unknown FoV Viewing Probabilities in Wireless Networks

Tile-size aware bitrate allocation for adaptive 360 video streaming

360Probdash

Optimal Viewport-Adaptive 360-Degree Video Streaming Against Random Head Movement.

JUST360: Optimizing 360-Degree Video Streaming Systems with Joint Utility

Mixed Tiling Scheme for Adaptive VR Streaming

Interactive Gigapixel Video Streaming Via Multiscale Acceleration