Abstract:360 images, with a field-of-view (FoV) of 180x360, provide immersive and realistic environments for emerging virtual reality (VR) applications, such as virtual tourism, where users desire to create diverse panoramic scenes from a narrow FoV photo they take from a viewpoint via portable devices. It thus brings us to a technical challenge: `How to allow the users to freely create diverse and immersive virtual scenes from a narrow FoV image with a specified viewport?' To this end, we propose a transformer-based 360 image outpainting framework called Dream360, which can generate diverse, high-fidelity, and high-resolution panoramas from user-selected viewports, considering the spherical properties of 360 images. Compared with existing methods, e.g., [3], which primarily focus on inputs with rectangular masks and central locations while overlooking the spherical property of 360 images, our Dream360 offers higher outpainting flexibility and fidelity based on the spherical representation. Dream360 comprises two key learning stages: (I) codebook-based panorama outpainting via Spherical-VQGAN (S-VQGAN), and (II) frequency-aware refinement with a novel frequency-aware consistency loss. Specifically, S-VQGAN learns a sphere-specific codebook from spherical harmonic (SH) values, providing a better representation of spherical data distribution for scene modeling. The frequency-aware refinement matches the resolution and further improves the semantic consistency and visual fidelity of the generated results. Our Dream360 achieves significantly lower Frechet Inception Distance (FID) scores and better visual fidelity than existing methods. We also conducted a user study involving 15 participants to interactively evaluate the quality of the generated results in VR, demonstrating the flexibility and superiority of our Dream360 framework.

Variable Aperture Panoramic Imaging

Design of Panoramic Stereo Imaging with Single Optical System

Mathematical Modeling and Implementation of a Panoramic Zoom System Without Central Blind Area

Panoramic Optical Annular Staring Imaging Technology

A Present and Development of Catadioptric Panoramic Stereo Imaging Technique

Research on Image Processing of Two-Dimensional Plan of the Hemispheric Panoramic Imaging System

Design of Vari-Focal Panoramic Annular Lenses Based on Alvarez Surfaces

Design of a Panoramic Annular Lens System with an Ultra-Wide Angle Via an Annular Gaussian Radial Basis Function Freeform Surface.

Design of a Compact Varifocal Panoramic System Based on the Mechanical Zoom Method.

Enhancement Of Panoramic Image Resolution Based On Swift Interpolation Of Bezier Surface

Omni-directional Image Unwarping Based on Taylor Model

Panoramic Imaging and 3D Reconstruction Based on Sparse Image Volume

Multi-Viewpoint Panorama Construction with Wide-Baseline Images

Panoramic 3 D Reconstruction Using Rotating Camera with Planar Mirrors

Freely Explore the Scene with 360°Field of View.

Panoramic 3d Reconstruction Using Stereo Multi-Perspective Panorama

Panoramic Stereo Imaging Using Hyperboloid Catadioptric Cameras

A Three Dimensional Measurement Approach and Experiment Based on Spherical Panorama

Dream360: Diverse and Immersive Outdoor Virtual Scene Creation via Transformer-Based 360 Image Outpainting

Three-dimensional Panoramic Sensor Based on a Variable-Frequency Fringe Pattern.

HORIZON: A High-Resolution Panorama Synthesis Framework