Abstract:360 images, with a field-of-view (FoV) of 180x360, provide immersive and realistic environments for emerging virtual reality (VR) applications, such as virtual tourism, where users desire to create diverse panoramic scenes from a narrow FoV photo they take from a viewpoint via portable devices. It thus brings us to a technical challenge: `How to allow the users to freely create diverse and immersive virtual scenes from a narrow FoV image with a specified viewport?' To this end, we propose a transformer-based 360 image outpainting framework called Dream360, which can generate diverse, high-fidelity, and high-resolution panoramas from user-selected viewports, considering the spherical properties of 360 images. Compared with existing methods, e.g., [3], which primarily focus on inputs with rectangular masks and central locations while overlooking the spherical property of 360 images, our Dream360 offers higher outpainting flexibility and fidelity based on the spherical representation. Dream360 comprises two key learning stages: (I) codebook-based panorama outpainting via Spherical-VQGAN (S-VQGAN), and (II) frequency-aware refinement with a novel frequency-aware consistency loss. Specifically, S-VQGAN learns a sphere-specific codebook from spherical harmonic (SH) values, providing a better representation of spherical data distribution for scene modeling. The frequency-aware refinement matches the resolution and further improves the semantic consistency and visual fidelity of the generated results. Our Dream360 achieves significantly lower Frechet Inception Distance (FID) scores and better visual fidelity than existing methods. We also conducted a user study involving 15 participants to interactively evaluate the quality of the generated results in VR, demonstrating the flexibility and superiority of our Dream360 framework.

Perceptual Analysis of Perspective Projection for Viewport Rendering in 360° Images

Globally and Locally Optimized Pannini Projection for High FoV Rendering of 360° Images

Globally and Locally Optimized Pannini Projection for High FoV Rendering of 360-degree Images

Visual Distortions in 360-degree Videos

VMP360

Viewport-adaptive navigable 360-degree video delivery

Quality Assessment of Stereoscopic 360-degree Images from Multi-viewports

Perceptual Quality Assessment of Omnidirectional Images: A Benchmark and Computational Model

Dream360: Diverse and Immersive Outdoor Virtual Scene Creation via Transformer-Based 360 Image Outpainting

An Overview of Panoramic Video Projection Schemes in the IEEE 1857.9 Standard for Immersive Visual Content Coding

Imagine360: Immersive 360 Video Generation from Perspective Anchor

Parallax 360 : Stereoscopic 360 ° Scene

VSOIQE: A Novel Viewport-Based Stitched 360° Omnidirectional Image Quality Evaluator

Freely Explore the Scene with 360°Field of View.

On the Effectiveness of Offset Projections for 360-Degree Video Streaming.

ResVR: Joint Rescaling and Viewport Rendering of Omnidirectional Images

Aximorphic Perspective Projection Model for Immersive Imagery

OmniSCV: An Omnidirectional Synthetic Image Generator for Computer Vision

Viewport-adaptive 360-degree video coding

Parallax360: Stereoscopic 360° Scene Representation for Head-Motion Parallax.

Perspective picture from Visual Sphere: a new approach to image rasterization