Abstract:Creating virtual reality (VR) content with effective imaging systems has attracted significant attention worldwide following the broad applications of VR in various fields, including entertainment, surveillance, sports, etc. However, due to the inherent trade-off between field-of-view and resolution of the imaging system as well as the prohibitive computational cost, live capturing and generating multiscale 360° 3D video content at an eye-limited resolution to provide immersive VR experiences confront significant challenges. In this work, we propose Multiscale-VR, a multiscale unstructured camera array computational imaging system for high-quality gigapixel 3D panoramic videography that creates the six-degree-of-freedom multiscale interactive VR content. The Multiscale-VR imaging system comprises scalable cylindrical-distributed global and local cameras, where global stereo cameras are stitched to cover 360° field-of-view, and unstructured local monocular cameras are adapted to the global camera for flexible high-resolution video streaming arrangement. We demonstrate that a high-quality gigapixel depth video can be faithfully reconstructed by our deep neural network-based algorithm pipeline where the global depth via stereo matching and the local depth via high-resolution RGB-guided refinement are associated. To generate the immersive 3D VR content, we present a three-layer rendering framework that includes an original layer for scene rendering, a diffusion layer for handling occlusion regions, and a dynamic layer for efficient dynamic foreground rendering. Our multiscale reconstruction architecture enables the proposed prototype system for rendering highly effective 3D, 360° gigapixel live VR video at 30 fps from the captured high-throughput multiscale video sequences. The proposed multiscale interactive VR content generation approach by using a heterogeneous camera system design, in contrast to the existing single-scale VR imaging systems with structured homogeneous cameras, will open up new avenues of research in VR and provide an unprecedented immersive experience benefiting various novel applications.

Multi-Camera System: Imaging Enhancement and Application

3D Framing Camera System Based on Sample Cumulation Method

Multi-image-distance Imaging System for Extending Depth-of-field

Joint Focusing Method Based on Optimal Image-Distance Interval for Multi-Image-distance Imaging System

Continuous Digital Zoom with Cross Attention for Dual Camera System

Research of real-time wide field image merging based on multi-cameras

Two-Stage Multi-Camera Constrain Mapping Pipeline for Large-Scale 3D Reconstruction

The Chameleon-Like Vision System

Hybrid-MVS: Robust Multi-View Reconstruction with Hybrid Optimization of Visual and Depth Cues

High resolution multispectral video capture with a hybrid camera system

Multi-spectral Imaging System Based on Light Field Rendering

Handheld Snapshot Multi-spectral Camera at 65-Megapixel Resolution

Multiscale-VR: Multiscale Gigapixel 3D Panoramic Videography for Virtual Reality

Acquisition of High Spatial and Spectral Resolution Video with a Hybrid Camera System

Scalable Multi-View Stereo Camera Array for Real-Time Image Capture and 3D Display in Real-World Applications

Monocular Multi-View Stereo Imaging System

A Flexible Multi-view Multi-modal Imaging System for Outdoor Scenes

Anatomy of a Multicamera Video Surveillance System

MCVO: A Generic Visual Odometry for Arbitrarily Arranged Multi-Cameras

Gigapixel imaging with a novel multi-camera array microscope

Research on multi-sensor high dynamic range imaging technology and application