PanoFree: Tuning-Free Holistic Multi-view Image Generation with Cross-view Self-Guidance

Aoming Liu,Zhong Li,Zhang Chen,Nannan Li,Yi Xu,Bryan A. Plummer

2024-08-05

Abstract:Immersive scene generation, notably panorama creation, benefits significantly from the adaptation of large pre-trained text-to-image (T2I) models for multi-view image generation. Due to the high cost of acquiring multi-view images, tuning-free generation is preferred. However, existing methods are either limited to simple correspondences or require extensive fine-tuning to capture complex ones. We present PanoFree, a novel method for tuning-free multi-view image generation that supports an extensive array of correspondences. PanoFree sequentially generates multi-view images using iterative warping and inpainting, addressing the key issues of inconsistency and artifacts from error accumulation without the need for fine-tuning. It improves error accumulation by enhancing cross-view awareness and refines the warping and inpainting processes via cross-view guidance, risky area estimation and erasing, and symmetric bidirectional guided generation for loop closure, alongside guidance-based semantic and density control for scene structure preservation. In experiments on Planar, 360°, and Full Spherical Panoramas, PanoFree demonstrates significant error reduction, improves global consistency, and boosts image quality without extra fine-tuning. Compared to existing methods, PanoFree is up to 5x more efficient in time and 3x more efficient in GPU memory usage, and maintains superior diversity of results (2x better in our user study). PanoFree offers a viable alternative to costly fine-tuning or the use of additional pre-trained models. Project website at <a class="link-external link-https" href="https://panofree.github.io/" rel="external noopener nofollow">this https URL</a>.

Computer Vision and Pattern Recognition

What problem does this paper attempt to address?

The paper attempts to address the problem of generating highly consistent and high-quality multi-view images, particularly in the task of panorama generation, without fine-tuning. Existing methods are either limited to simple correspondences or require extensive fine-tuning to capture complex correspondences, which not only increases data and computational costs but also limits the scope of applications. PanoFree proposes a fine-tuning-free multi-view image generation method that supports a wide range of correspondences through iterative deformation and repair techniques, significantly reducing error accumulation, improving global consistency, and maintaining result diversity.

Specifically, PanoFree addresses the following issues:
1. **Error Accumulation**: Existing methods are prone to error accumulation during the generation process, leading to a decline in image quality and inconsistency.
2. **Global Consistency**: The generated multi-view images have poor global consistency, especially in panorama generation.
3. **Computational Efficiency**: Existing methods require a large amount of time and memory resources during the generation process.
4. **Diversity**: The generated images perform poorly in terms of diversity, especially when using the same text prompts.

PanoFree addresses these issues through the following technical means:
- **Cross-View Guidance**: Utilizes images from multiple views as conditions to enhance cross-view perception and reduce bias.
- **Risk Area Estimation and Erasure**: Identifies and erases areas that may lead to errors, reducing noise propagation.
- **Bidirectional Generation and Symmetric Guidance**: Adopts a bidirectional generation path and ensures loop closure through symmetric guidance, reducing errors.
- **Guided Semantic and Density Control**: Controls semantics and density to make the scene structure more reasonable.

Through these technical means, PanoFree demonstrates significant advantages in multiple experiments, including higher image quality and global consistency, lower computational costs, and better result diversity.