4K4DGen: Panoramic 4D Generation at 4K Resolution

Renjie Li,Panwang Pan,Bangbang Yang,Dejia Xu,Shijie Zhou,Xuanyang Zhang,Zeming Li,Achuta Kadambi,Zhangyang Wang,Zhengzhong Tu,Zhiwen Fan

2024-10-03

Abstract:The blooming of virtual reality and augmented reality (VR/AR) technologies has driven an increasing demand for the creation of high-quality, immersive, and dynamic environments. However, existing generative techniques either focus solely on dynamic objects or perform outpainting from a single perspective image, failing to meet the requirements of VR/AR applications that need free-viewpoint, 360$^{\circ}$ virtual views where users can move in all directions. In this work, we tackle the challenging task of elevating a single panorama to an immersive 4D experience. For the first time, we demonstrate the capability to generate omnidirectional dynamic scenes with 360$^{\circ}$ views at 4K (4096 $\times$ 2048) resolution, thereby providing an immersive user experience. Our method introduces a pipeline that facilitates natural scene animations and optimizes a set of dynamic Gaussians using efficient splatting techniques for real-time exploration. To overcome the lack of scene-scale annotated 4D data and models, especially in panoramic formats, we propose a novel \textbf{Panoramic Denoiser} that adapts generic 2D diffusion priors to animate consistently in 360$^{\circ}$ images, transforming them into panoramic videos with dynamic scenes at targeted regions. Subsequently, we propose \textbf{Dynamic Panoramic Lifting} to elevate the panoramic video into a 4D immersive environment while preserving spatial and temporal consistency. By transferring prior knowledge from 2D models in the perspective domain to the panoramic domain and the 4D lifting with spatial appearance and geometry regularization, we achieve high-quality Panorama-to-4D generation at a resolution of 4K for the first time.

Computer Vision and Pattern Recognition

What problem does this paper attempt to address?

The main problem that this paper attempts to solve is to generate high - quality, immersive 4D panoramic environments to meet the requirements of virtual reality (VR) and augmented reality (AR) applications. Specifically, the existing generation techniques either focus only on dynamic objects or extrapolate from single - view images, and cannot meet the requirements of VR/AR applications that require free - view and 360° virtual views. Therefore, this paper proposes a brand - new framework - 4K4DGen, which aims to convert static panoramic images into dynamic 4D scenes with high resolution (4K, that is, 4096×2048), thereby providing an immersive user experience.

### Main Problems and Challenges
1. **Lack of large - scale labeled 4D data**: Especially in the panoramic format, which restricts the training of specialized models.
2. **Maintaining global consistency and local details**: It is very difficult for existing 2D diffusion models to simultaneously achieve fine - grained local details and global consistency in 4D and 4K panoramic views.

### Solution Overview
- **Panoramic Denoiser**: Ensure the consistent animation of panoramic videos by denoising spherical latent codes, adapt to the pre - trained 2D view - angle diffusion model, and generate 360° panoramic images with dynamic scene elements.
- **Dynamic Panoramic Lifting**: Promote dynamic panoramic videos to 4D environmental assets, use a set of dynamic Gaussian distributions, and combine spatio - temporal geometric alignment to ensure cross - frame consistency.

### Specific Methods
1. **Panoramic Denoiser**:
- Map the static panorama to the spherical latent space.
- In each denoising step, project the spherical latent code into multiple view - angle latent codes, and use the pre - trained view - angle denoiser to gradually remove noise.
- Optimize the spherical latent code by fusing all the denoised view - angle latent codes to ensure global consistency.

2. **Dynamic Panoramic Lifting**:
- Use a monocular depth estimator to generate a depth map for each frame, and fuse it into a coherent panoramic depth map through spatio - temporal geometric alignment.
- Represent the dynamic scene using 3D Gaussian distributions, and optimize these Gaussian distributions through supervised learning to ensure that the rendering results are consistent with the input panoramic video.

### Experimental Verification
- **Experimental Setup**: Conduct experiments using NVIDIA A100 GPU, select different camera orientations and parameters, and evaluate the quality of the generated 4D scenes.
- **Evaluation Method**: Since there is no real - data of 4D scenes, the author renders videos of specific test camera poses through synthetic 4D representations and uses no - reference video / image quality evaluation methods for quantitative evaluation.

In summary, this paper successfully solves the key challenges of converting static panoramic images into high - quality, immersive 4D environments by proposing the 4K4DGen framework, providing new solutions for VR/AR applications.