Compositional Image Decomposition with Diffusion Models

Jocelin Su,Nan Liu,Yanbo Wang,Joshua B. Tenenbaum,Yilun Du

2024-06-28

Abstract:Given an image of a natural scene, we are able to quickly decompose it into a set of components such as objects, lighting, shadows, and foreground. We can then envision a scene where we combine certain components with those from other images, for instance a set of objects from our bedroom and animals from a zoo under the lighting conditions of a forest, even if we have never encountered such a scene before. In this paper, we present a method to decompose an image into such compositional components. Our approach, Decomp Diffusion, is an unsupervised method which, when given a single image, infers a set of different components in the image, each represented by a diffusion model. We demonstrate how components can capture different factors of the scene, ranging from global scene descriptors like shadows or facial expression to local scene descriptors like constituent objects. We further illustrate how inferred factors can be flexibly composed, even with factors inferred from other models, to generate a variety of scenes sharply different than those seen in training time. Website and code at <a class="link-external link-https" href="https://energy-based-model.github.io/decomp-diffusion" rel="external noopener nofollow">this https URL</a>.

Computer Vision and Pattern Recognition,Machine Learning

What problem does this paper attempt to address?

The problem that this paper attempts to solve is how to decompose a set of components, such as objects, illumination, shadows, and foregrounds, from images of natural scenes, and be able to flexibly combine these components to generate new scenes. Specifically, the paper proposes a method named Decomp Diffusion, which is an unsupervised method that can infer different components from a single image, with each component represented by a diffusion model. This method can not only capture different factors of the scene, including global scene descriptors (such as shadows or facial expressions) and local scene descriptors (such as constituent objects), but also generate various scenes significantly different from those seen during training by combining the factors inferred by different models.

The main contributions of the paper include:
1. Proposing the Decomp Diffusion method, which uses diffusion models to decompose scenes into different constituent concepts, significantly outperforming previous work using explicit energy functions.
2. Demonstrating that Decomp Diffusion can successfully decompose scenes into global and local concepts.
3. Proving that the concepts discovered by Decomp Diffusion have good generalization ability and can be combined across different data modalities and components discovered by Decomp Diffusion of other instances.