Abstract:Panoramic images provide comprehensive scene information and are suitable for VR applications. Obtaining corresponding depth maps is essential for achieving immersive and interactive experiences. However, panoramic depth estimation presents significant challenges due to the severe distortion caused by equirectangular projection (ERP) and the limited availability of panoramic RGB-D datasets. Inspired by the recent success of neural rendering, we propose a novel method, named $\mathbf{CUBE360}$, that learns a cubic field composed of multiple MPIs from a single panoramic image for $\mathbf{continuous}$ depth estimation at any view direction. Our CUBE360 employs cubemap projection to transform an ERP image into six faces and extract the MPIs for each, thereby reducing the memory consumption required for MPI processing of high-resolution data. Additionally, this approach avoids the computational complexity of handling the uneven pixel distribution inherent to equirectangular projectio. An attention-based blending module is then employed to learn correlations among the MPIs of cubic faces, constructing a cubic field representation with color and density information at various depth levels. Furthermore, a novel sampling strategy is introduced for rendering novel views from the cubic field at both cubic and planar scales. The entire pipeline is trained using photometric loss calculated from rendered views within a self-supervised learning approach, enabling training on 360 videos without depth annotations. Experiments on both synthetic and real-world datasets demonstrate the superior performance of CUBE360 compared to prior SSL methods. We also highlight its effectiveness in downstream applications, such as VR roaming and visual effects, underscoring CUBE360's potential to enhance immersive experiences.

SPCNet: A Panoramic Image Depth Estimation Method Based on Spherical Convolution

A Robust Monocular Depth Estimation Framework Based on Light-Weight ERF-Pspnet for Day-Night Driving Scenes

SphereDepth: Panorama Depth Estimation from Spherical Domain

Binocular Depth Estimation Using Convolutional Neural Network With Siamese Branches.

Real-time Depth Estimation for Aerial Panoramas in Virtual Reality.

PADENet: an Efficient and Robust Panoramic Monocular Depth Estimation Network for Outdoor Scenes.

Afnet: Asymmetric Fusion Network for Monocular Panorama Depth Estimation

SPDET: Edge-Aware Self-Supervised Panoramic Depth Estimation Transformer With Spherical Geometry

Omnidirectional stereo depth estimation based on spherical deep network

E2LNet: An Efficient and Effective Lightweight Network for Panoramic Depth Estimation

ACDNet: Adaptively Combined Dilated Convolution for Monocular Panorama Depth Estimation

Panoramic Image Depth Estimation Based on Deformable Convolution and Attention Mechanism

CUBE360: Learning Cubic Field Representation for Monocular 360 Depth Estimation for Virtual Reality

GLPanoDepth: Global-to-Local Panoramic Depth Estimation

DPS-Net: Deep Polarimetric Stereo Depth Estimation

MSI-NeRF: Linking Omni-Depth with View Synthesis through Multi-Sphere Image aided Generalizable Neural Radiance Field

Depth Estimation from Indoor Panoramas with Neural Scene Representation

Panoramic Depth Estimation Algorithm Based on Attention Module

PanoFormer: Panorama Transformer for Indoor 360 Depth Estimation

CRF360D: Monocular 360 Depth Estimation via Spherical Fully-Connected CRFs

Panoramic Imaging and 3D Reconstruction Based on Sparse Image Volume