Abstract:Creating virtual reality (VR) content with effective imaging systems has attracted significant attention worldwide following the broad applications of VR in various fields, including entertainment, surveillance, sports, etc. However, due to the inherent trade-off between field-of-view and resolution of the imaging system as well as the prohibitive computational cost, live capturing and generating multiscale 360° 3D video content at an eye-limited resolution to provide immersive VR experiences confront significant challenges. In this work, we propose Multiscale-VR, a multiscale unstructured camera array computational imaging system for high-quality gigapixel 3D panoramic videography that creates the six-degree-of-freedom multiscale interactive VR content. The Multiscale-VR imaging system comprises scalable cylindrical-distributed global and local cameras, where global stereo cameras are stitched to cover 360° field-of-view, and unstructured local monocular cameras are adapted to the global camera for flexible high-resolution video streaming arrangement. We demonstrate that a high-quality gigapixel depth video can be faithfully reconstructed by our deep neural network-based algorithm pipeline where the global depth via stereo matching and the local depth via high-resolution RGB-guided refinement are associated. To generate the immersive 3D VR content, we present a three-layer rendering framework that includes an original layer for scene rendering, a diffusion layer for handling occlusion regions, and a dynamic layer for efficient dynamic foreground rendering. Our multiscale reconstruction architecture enables the proposed prototype system for rendering highly effective 3D, 360° gigapixel live VR video at 30 fps from the captured high-throughput multiscale video sequences. The proposed multiscale interactive VR content generation approach by using a heterogeneous camera system design, in contrast to the existing single-scale VR imaging systems with structured homogeneous cameras, will open up new avenues of research in VR and provide an unprecedented immersive experience benefiting various novel applications.

A Survey on 360° Images and Videos in Mixed Reality: Algorithms and Applications

A Survey on 360° Video Streaming

A Survey on 360° Video Streaming: Acquisition, Transmission, and Display

Survey on Light Field 3D Display Applied to Mixed Reality

360-Degree Video Streaming: A Survey of the State of the Art

Application Research of Virtual Reality Technology in Film and Television Technology

A Survey of Monocular Simultaneous Localization and Mapping

VR content creation and exploration with deep learning: A survey

State-of-the-Art in 360° Video/Image Processing: Perception, Assessment and Compression

From Capture to Display: A Survey on Volumetric Video

A Survey on Multiview Video Synthesis and Editing

Recent Advances in Immersive Multimedia

A review of research on 360-degree video and its applications to education

MatryODShka: Real-time 6DoF Video View Synthesis using Multi-Sphere Images

Multiscale-VR: Multiscale Gigapixel 3D Panoramic Videography for Virtual Reality

Mapping the Viewer Experience in Cinematic Virtual Reality: A Systematic Review

A Survey on Metaverse: Technologies, Applications, Future Trends and Challenges

AI-Enhanced Virtual Reality in Medicine: A Comprehensive Survey

Collaborative MR Workspace with Shared 3D Vision Based on Stereo Video Transmission

3D Video Representation and Design for Ubiquitous Environments