Abstract:3D content creation from a single image is a long-standing yet highly desirable task. Recent advances introduce 2D diffusion priors, yielding reasonable results. However, existing methods are not hyper-realistic enough for post-generation usage, as users cannot view, render and edit the resulting 3D content from a full range. To address these challenges, we introduce HyperDreamer with several key designs and appealing properties: 1) Viewable: 360 degree mesh modeling with high-resolution textures enables the creation of visually compelling 3D models from a full range of observation points. 2) Renderable: Fine-grained semantic segmentation and data-driven priors are incorporated as guidance to learn reasonable albedo, roughness, and specular properties of the materials, enabling semantic-aware arbitrary material estimation. 3) Editable: For a generated model or their own data, users can interactively select any region via a few clicks and efficiently edit the texture with text-based guidance. Extensive experiments demonstrate the effectiveness of HyperDreamer in modeling region-aware materials with high-resolution textures and enabling user-friendly editing. We believe that HyperDreamer holds promise for advancing 3D content creation and finding applications in various domains.

What problem does this paper attempt to address?

This paper attempts to solve the problem of generating and editing surreal 3D content from a single image. Specifically, existing 3D content generation methods have two main problems: 1. **Limited post - generation usability**: Current methods usually use implicit 3D representations, which sacrifice usability while improving fidelity. Users cannot freely scale, re - render or edit the generated 3D content, which limits its practical applications and creative potential. 2. **2D diffusion bias**: These methods rely on diffusion models trained on 2D datasets with rich lighting and shadow variations. Although these variations enhance the realism of 2D images, they also cause unwanted effects in the textures of 3D models (as shown in Figure 4 - d). To solve these problems, the authors propose the HyperDreamer framework, which has the following three key features: 1. **Full - range viewability**: By introducing a novel custom super - resolution module, HyperDreamer can generate high - resolution textures and support viewing 3D models from any viewing angle. 2. **Full - range renderability**: Combining online 3D semantic segmentation and spatially - varying bidirectional reflectance distribution function (BRDF), HyperDreamer can learn the reasonable albedo, roughness and specular reflection properties of materials, thus achieving more realistic rendering. 3. **Full - range editability**: HyperDreamer allows users to select specific areas by simple clicking and efficiently edit textures based on text guidance, greatly improving the flexibility and ease of use of editing. Through these improvements, HyperDreamer not only significantly improves the quality of 3D content generation, but also expands the accessibility and practicality of AI - generated 3D content in practical applications.

HyperDreamer: Hyper-Realistic 3D Content Generation and Editing from a Single Image

EfficientDreamer: High-Fidelity and Robust 3D Creation via Orthogonal-view Diffusion Prior

IPDreamer: Appearance-Controllable 3D Object Generation with Complex Image Prompts

Novel 3D-Aware Composition Images Synthesis for Object Display with Diffusion Model.

Creating High-quality 3D Content by Bridging the Gap Between Text-to-2D and Text-to-3D Generation

IPDreamer: Appearance-Controllable 3D Object Generation with Image Prompts.

DreamCraft3D: Hierarchical 3D Generation with Bootstrapped Diffusion Prior

X-Dreamer: Creating High-quality 3D Content by Bridging the Domain Gap Between Text-to-2D and Text-to-3D Generation

MetaDreamer: Efficient Text-to-3D Creation With Disentangling Geometry and Texture

HoloDreamer: Holistic 3D Panoramic World Generation from Text Descriptions

Magic3D: High-Resolution Text-to-3D Content Creation

Enhanced 3D Generation by 2D Editing

ImageDream: Image-Prompt Multi-view Diffusion for 3D Generation

BoostDream: Efficient Refining for High-Quality Text-to-3D Generation from Multi-View Diffusion

TextureDreamer: Image-guided Texture Synthesis through Geometry-aware Diffusion

High-Fidelity 3D Model Generation with Relightable Appearance from Single Freehand Sketches and Text Guidance

Make-It-3D: High-Fidelity 3D Creation from A Single Image with Diffusion Prior

RealmDreamer: Text-Driven 3D Scene Generation with Inpainting and Depth Diffusion

3D Creation at Your Fingertips: From Text or Image to 3D Assets

EucliDreamer: Fast and High-Quality Texturing for 3D Models with Depth-Conditioned Stable Diffusion

GeoDream: Disentangling 2D and Geometric Priors for High-Fidelity and Consistent 3D Generation