Abstract:Immersive video applications grow faster for users to freely navigate within a virtualized 3D environment for entertainment, productivity, training, etc. Fundamentally, such system can be facilitated by an interactive Gigapixel Video Streaming (iGVS) platform from array camera capturing to end user interaction. This interactive system demands a large amount of network bandwidth to sustain the reliable service provisioning, hindering its massive market adoption. Thus, we propose to segment the gigapixel scene into non-overlapped spatial tiles. Each tile only covers a sub-region of the entire scene. One or more tiles will be used to represent an instantaneous viewport interested by a specific user. Tiles are then encoded at a variety of quality scales using various combinations of spatial, temporal and amplitude resolutions (STAR), which are typically encapsulated into temporally-aligned tile video chunks (or simply chunks). Chunks at different quality level can be processed in parallel for real-time purpose. With such setup, diverse chunk combinations can be simultaneously accessed by heterogeneous user per its request, and viewport-adaptation based content navigation in an immersive space can be also realized by adapting multiscale chunks properly, under the bandwidth constraints. A serial computational vision models measuring the perceptual quality of viewport video in terms of its quality scales, adaptation factors, as well as the peripheral vision thresholds, are devised to prepare and guide the chunk adaptation for the best perceptual quality index. Furthermore, in response to the time-varying network, a deep reinforcement learning (DRL) based adaptive real-time streaming (ARS) scheme is developed, by learning the future decision from the historical network states, to maximize the overall quality of experience (QoE) in a practical Internet-based streaming scenario. Our experiments have revealed that averaged QoE can be improved by about 60%, and its standard deviation can be also reduced by approximate to 30%, in comparison to the popular Google congestion control algorithm widely adopted in existing system for adaptive streaming, demonstrating the efficiency of our multiscale accelerated iGVS for immersive video application.

The On-Line Viewing System of the Distributed Gigapixel Image

Real-time viewing of large images based on multi-core

Large-Scale Model Rendering System Based on Network Streaming

Real-time browsing of high-precision frescos

FSIS: Fast and Seamless Image Stitching of High-Resolution Dunhuang Murals

Interactive Browsing of Large Images on Multi-Projector Display Wall System

Large area interactive browsing for high resolution digitized dunhuang murals

Web-based virtual walkthrough of panoramas

Universal Browse and Display Framework for Ultra-Resolution Image

A Mechanism of Remote Sensing Image for Parallel Processing Base on Splitting Blocks

Parallel Visualization for Large-Scale Datasets

Research on remote sensing image storage management and a fast visualization system based on cloud computing technology

Balancing Cpu And Gpu: Real-Time Visualization Of Large Scale 3d Scanning Models

DCPMS: A Large-Scale Raster Layer Serving Method for Custom Online Calculation and Rendering

Multi-screen Tiled Displayed, Parallel Rendering System for a Large Terrain Dataset

A Parallel Computing System Of Remote Sensing Images Based On Global Subdivision Model

Interactive Gigapixel Video Streaming Via Multiscale Acceleration

Interactive Rendering for Large-Scale Mesh Based on MapReduce

Progressive Image Retrieval with Quality Guarantee under MapReduce Framework

Parallel Multigrid Approach for Solving Poisson PDE in Gigapixel Image Editing

Efficient Management and Scheduling of Massive Remote Sensing Image Datasets