Abstract:360 images, with a field-of-view (FoV) of 180x360, provide immersive and realistic environments for emerging virtual reality (VR) applications, such as virtual tourism, where users desire to create diverse panoramic scenes from a narrow FoV photo they take from a viewpoint via portable devices. It thus brings us to a technical challenge: `How to allow the users to freely create diverse and immersive virtual scenes from a narrow FoV image with a specified viewport?' To this end, we propose a transformer-based 360 image outpainting framework called Dream360, which can generate diverse, high-fidelity, and high-resolution panoramas from user-selected viewports, considering the spherical properties of 360 images. Compared with existing methods, e.g., [3], which primarily focus on inputs with rectangular masks and central locations while overlooking the spherical property of 360 images, our Dream360 offers higher outpainting flexibility and fidelity based on the spherical representation. Dream360 comprises two key learning stages: (I) codebook-based panorama outpainting via Spherical-VQGAN (S-VQGAN), and (II) frequency-aware refinement with a novel frequency-aware consistency loss. Specifically, S-VQGAN learns a sphere-specific codebook from spherical harmonic (SH) values, providing a better representation of spherical data distribution for scene modeling. The frequency-aware refinement matches the resolution and further improves the semantic consistency and visual fidelity of the generated results. Our Dream360 achieves significantly lower Frechet Inception Distance (FID) scores and better visual fidelity than existing methods. We also conducted a user study involving 15 participants to interactively evaluate the quality of the generated results in VR, demonstrating the flexibility and superiority of our Dream360 framework.

WonderJourney: Going from Anywhere to Everywhere

WonderWorld: Interactive 3D Scene Generation from a Single Image

Web-based virtual walkthrough of panoramas

DreamScene360: Unconstrained Text-to-3D Scene Generation with Panoramic Gaussian Splatting

Persistent Nature: A Generative Model of Unbounded 3D Worlds

HoloDreamer: Holistic 3D Panoramic World Generation from Text Descriptions

SceneDreamer360: Text-Driven 3D-Consistent Scene Generation with Panoramic Gaussian Splatting

The Wanderings of Odysseus in 3D Scenes

SceneDreamer: Unbounded 3D Scene Generation from 2D Image Collections

Unrolling Virtual Worlds for Immersive Experiences

BerfScene: Bev-conditioned Equivariant Radiance Fields for Infinite 3D Scene Generation

VividDream: Generating 3D Scene with Ambient Dynamics

Director3D: Real-world Camera Trajectory and 3D Scene Generation from Text

LayerPano3D: Layered 3D Panorama for Hyper-Immersive Scene Generation

OccSora: 4D Occupancy Generation Models as World Simulators for Autonomous Driving

Generating Visual Spatial Description via Holistic 3D Scene Understanding

Text2Immersion: Generative Immersive Scene with 3D Gaussians

Dream360: Diverse and Immersive Outdoor Virtual Scene Creation via Transformer-Based 360 Image Outpainting

SceneCraft: Layout-Guided 3D Scene Generation

Urban Architect: Steerable 3D Urban Scene Generation with Layout Prior

RealmDreamer: Text-Driven 3D Scene Generation with Inpainting and Depth Diffusion