Abstract:360 images, with a field-of-view (FoV) of 180x360, provide immersive and realistic environments for emerging virtual reality (VR) applications, such as virtual tourism, where users desire to create diverse panoramic scenes from a narrow FoV photo they take from a viewpoint via portable devices. It thus brings us to a technical challenge: `How to allow the users to freely create diverse and immersive virtual scenes from a narrow FoV image with a specified viewport?' To this end, we propose a transformer-based 360 image outpainting framework called Dream360, which can generate diverse, high-fidelity, and high-resolution panoramas from user-selected viewports, considering the spherical properties of 360 images. Compared with existing methods, e.g., [3], which primarily focus on inputs with rectangular masks and central locations while overlooking the spherical property of 360 images, our Dream360 offers higher outpainting flexibility and fidelity based on the spherical representation. Dream360 comprises two key learning stages: (I) codebook-based panorama outpainting via Spherical-VQGAN (S-VQGAN), and (II) frequency-aware refinement with a novel frequency-aware consistency loss. Specifically, S-VQGAN learns a sphere-specific codebook from spherical harmonic (SH) values, providing a better representation of spherical data distribution for scene modeling. The frequency-aware refinement matches the resolution and further improves the semantic consistency and visual fidelity of the generated results. Our Dream360 achieves significantly lower Frechet Inception Distance (FID) scores and better visual fidelity than existing methods. We also conducted a user study involving 15 participants to interactively evaluate the quality of the generated results in VR, demonstrating the flexibility and superiority of our Dream360 framework.

PanoDreamer: 3D Panorama Synthesis from a Single Image

DeepPanoContext: Panoramic 3D Scene Understanding with Holistic Scene Context Graph and Relation-based Optimization

Multi-Viewpoint Panorama Construction with Wide-Baseline Images

Panoramic 3d Reconstruction Using Stereo Multi-Perspective Panorama

Omnidirectional 3D Reconstruction Using Stereo Multi-Perspective Panoramas

Panoramic Imaging and 3D Reconstruction Based on Sparse Image Volume

Stereo Reconstruction from Multiperspective Panoramas

HORIZON: A High-Resolution Panorama Synthesis Framework

Dream360: Diverse and Immersive Outdoor Virtual Scene Creation via Transformer-Based 360 Image Outpainting

Panoramic 3D Reconstruction Using Rotational Stereo Camera with Simple Epipolar Constraints

Panoramic 3 D Reconstruction Using Rotating Camera with Planar Mirrors

PanoDepth: A Two-Stage Approach for Monocular Omnidirectional Depth Estimation

Fast Generation Of Dynamic And Multi-Resolution 360 Degrees Panorama From Video Sequences

HORIZON: High-Resolution Semantically Controlled Panorama Synthesis

Construction 3 D Panoramic Model of Natural Scene from Real Image Sequences

Fast Construction of Dynamic and Multi-Resolution 360° Panoramas from Video Sequences

HoloDreamer: Holistic 3D Panoramic World Generation from Text Descriptions

DreamScene360: Unconstrained Text-to-3D Scene Generation with Panoramic Gaussian Splatting

LayerPano3D: Layered 3D Panorama for Hyper-Immersive Scene Generation

Pano2Room: Novel View Synthesis from a Single Indoor Panorama

360-Degree Panorama Generation from Few Unregistered NFoV Images