PartCraft: Crafting Creative Objects by Parts

Kam Woh Ng,Xiatian Zhu,Yi-Zhe Song,Tao Xiang

2024-07-08

Abstract:This paper propels creative control in generative visual AI by allowing users to "select". Departing from traditional text or sketch-based methods, we for the first time allow users to choose visual concepts by parts for their creative endeavors. The outcome is fine-grained generation that precisely captures selected visual concepts, ensuring a holistically faithful and plausible result. To achieve this, we first parse objects into parts through unsupervised feature clustering. Then, we encode parts into text tokens and introduce an entropy-based normalized attention loss that operates on them. This loss design enables our model to learn generic prior topology knowledge about object's part composition, and further generalize to novel part compositions to ensure the generation looks holistically faithful. Lastly, we employ a bottleneck encoder to project the part tokens. This not only enhances fidelity but also accelerates learning, by leveraging shared knowledge and facilitating information exchange among instances. Visual results in the paper and supplementary material showcase the compelling power of PartCraft in crafting highly customized, innovative creations, exemplified by the "charming" and creative birds. Code is released at <a class="link-external link-https" href="https://github.com/kamwoh/partcraft" rel="external noopener nofollow">this https URL</a>.

Computer Vision and Pattern Recognition

What problem does this paper attempt to address?

### Problems Addressed by the Paper

The paper primarily addresses the issue of "control" in generative AI, specifically how to enable users to have more fine-grained control over the generation process. Currently, most generative AI models (such as text-to-image models) lack sufficient user control during the generation process, especially in terms of details. For example, when a bird enthusiast wants to create a unique bird, existing methods either fail to accurately achieve the user's intent (e.g., through text prompts alone) or require complex inputs (such as additional control mechanisms, including semantic segmentation masks or bounding boxes).

#### Main Contributions
1. **Fine-Grained Control**: The paper proposes a new method that allows users to generate objects by selecting specific parts, thereby achieving fine-grained control over the generation process. This method not only simplifies user involvement but also ensures that the generated objects have overall consistency and coherence.
2. **PartCraft Technology**: By automatically parsing parts of objects and combining these parts from different visual concepts, a new object is generated. This method allows users to achieve creative generation by simply selecting the desired parts.
3. **Attention Loss and Bottleneck Encoder**: To enhance part decoupling and generation fidelity, the paper introduces an entropy-based normalized attention loss and uses a bottleneck encoder. This helps the model better understand the relationships between object parts, resulting in more natural and coherent objects.
4. **Quantitative Evaluation Metrics**: Two problem-specific quantitative evaluation metrics (EMR and CoSim) are proposed, and experiments validate the superior performance of the proposed method in generating novel objects.

In summary, the paper aims to improve the controllability and creativity of generative AI models through an innovative approach, allowing users to more flexibly realize their creative ideas.