Abstract:Novel view synthesis is a fundamental challenge in image-to-3D generation, requiring the generation of target view images from a set of conditioning images and their relative poses. While recent approaches like Zero-1-to-3 have demonstrated promising results using conditional latent diffusion models, they face significant challenges in generating consistent and accurate novel views, particularly when handling multiple conditioning images. In this work, we conduct a thorough investigation of Zero-1-to-3's cross-attention mechanism within the Spatial Transformer of the diffusion 2D-conditional UNet. Our analysis reveals a critical discrepancy between Zero-1-to-3's theoretical framework and its implementation, specifically in the processing of image-conditional context. We propose two significant improvements: (1) a corrected implementation that enables effective utilization of the cross-attention mechanism, and (2) an enhanced architecture that can leverage multiple conditional views simultaneously. Our theoretical analysis and preliminary results suggest potential improvements in novel view synthesis consistency and accuracy.

What problem does this paper attempt to address?

### What problems does this paper attempt to solve? This paper mainly explores and attempts to solve the problems encountered in image - to - 3D generation, especially in the novel view synthesis task. Specifically: 1. **Limitations of existing models**: - The paper points out that although recent methods such as Zero - 1 - to - 3 have shown promising results using conditional latent diffusion models, when dealing with multiple conditional images, these methods face significant challenges in generating consistent and accurate new views. - In particular, there are key differences between theory and practice in the implementation of the cross - attention mechanism of Zero - 1 - to - 3 in its Spatial Transformer, resulting in poor processing of image - conditional contexts. 2. **Defects of the cross - attention mechanism**: - Through analysis, the paper finds that when the cross - attention mechanism of Zero - 1 - to - 3 processes image - conditional contexts, its attention weights degenerate into a 1D vector with uniform values, thus limiting its ability to process conditional information. This problem makes the model unable to effectively utilize multi - view conditional information. 3. **New view synthesis under multi - view conditions**: - Existing single - view conditional generation methods perform poorly when generating the back - view of an object. Therefore, it is necessary to explore how to use multiple input views and their corresponding camera angles to provide richer spatial information and ensure view consistency. ### Proposed solutions To address the above problems, the paper proposes the following improvement measures: 1. **Improved cross - attention mechanism**: - A modified implementation method is proposed to enable the cross - attention mechanism to use conditional information more effectively. 2. **Enhanced architecture design**: - A new architecture is designed that can simultaneously use multiple conditional views in UNet to improve the consistency and accuracy of new view synthesis. 3. **Multi - view conditional generation**: - By introducing multi - view encoding paths and cross - attention integration modules, information from multiple views is fused into a unified representation, thereby better handling complex object geometries. 4. **Redesigned embedding method**: - The embedding methods of image and angle conditions are improved. The image conditions and angle conditions are directly encoded into the same dimension and vertically concatenated so that the cross - attention mechanism can fully utilize the provided context information. ### Summary This paper aims to improve the quality and consistency of new view synthesis under multi - view conditions, especially when dealing with complex object geometries, by in - depth analysis of the cross - attention mechanism of existing models and proposing improvement schemes.

Fixing the Perspective: A Critical Examination of Zero-1-to-3

Zero-1-to-3: Zero-shot One Image to 3D Object

Zero-to-Hero: Enhancing Zero-Shot Novel View Synthesis via Attention Map Filtering

Cascade-Zero123: One Image to Highly Consistent 3D with Self-Prompted Nearby Views

ZeroNVS: Zero-Shot 360-Degree View Synthesis from a Single Image

Consistent-1-to-3: Consistent Image to 3D View Synthesis via Geometry-aware Diffusion Models

Zero123++: a Single Image to Consistent Multi-view Diffusion Base Model

DiffPortrait3D: Controllable Diffusion for Zero-Shot Portrait View Synthesis

Transformer-Based Approach Via Contrastive Learning for Zero-Shot Detection.

Pragmatist: Multiview Conditional Diffusion Models for High-Fidelity 3D Reconstruction from Unposed Sparse Views

Efficient-3DiM: Learning a Generalizable Single-image Novel-view Synthesizer in One Day

Consistent123: Improve Consistency for One Image to 3D Object Synthesis

ViVid-1-to-3: Novel View Synthesis with Video Diffusion Models

The More You See in 2D, the More You Perceive in 3D

Zero123-6D: Zero-shot Novel View Synthesis for RGB Category-level 6D Pose Estimation

Long-Term Photometric Consistent Novel View Synthesis with Diffusion Models

AUTO3D: Novel view synthesis through unsupervisely learned variational viewpoint and global 3D representation

Ctrl123: Consistent Novel View Synthesis via Closed-Loop Transcription

Unaligned 2D to 3D Translation with Conditional Vector-Quantized Code Diffusion using Transformers

SparseFusion: Distilling View-conditioned Diffusion for 3D Reconstruction