Abstract:Humans leverage compositionality to efficiently learn new concepts, understanding how familiar parts can combine together to form novel objects. In contrast, popular computer vision models struggle to make the same types of inferences, requiring more data and generalizing less flexibly than people do. Here, we study these distinctively human abilities across a range of different types of visual composition, examining how people classify and generate ``alien figures'' with rich relational structure. We also develop a Bayesian program induction model which searches for the best programs for generating the candidate visual figures, utilizing a large program space containing different compositional mechanisms and abstractions. In few shot classification tasks, we find that people and the program induction model can make a range of meaningful compositional generalizations, with the model providing a strong account of the experimental data as well as interpretable parameters that reveal human assumptions about the factors invariant to category membership (here, to rotation and changing part attachment). In few shot generation tasks, both people and the models are able to construct compelling novel examples, with people behaving in additional structured ways beyond the model capabilities, e.g. making choices that complete a set or reconfiguring existing parts in highly novel ways. To capture these additional behavioral patterns, we develop an alternative model based on neuro-symbolic program induction: this model also composes new concepts from existing parts yet, distinctively, it utilizes neural network modules to successfully capture residual statistical structure. Together, our behavioral and computational findings show how people and models can produce a rich variety of compositional behavior when classifying and generating visual objects.

Compositional Abilities Emerge Multiplicatively: Exploring Diffusion Models on a Synthetic Task

How Diffusion Models Learn to Factorize and Compose

Dynamics of Concept Learning and Compositional Generalization

Reduce, Reuse, Recycle: Compositional Generation with Energy-Based Diffusion Models and MCMC

Progressive Compositionality In Text-to-Image Generative Models

Compositional diversity in visual concept learning

Compositional Generative Modeling: A Single Model is Not All You Need

Composer: Creative and Controllable Image Synthesis with Composable Conditions

IterComp: Iterative Composition-Aware Feedback Learning from Model Gallery for Text-to-Image Generation

Compositionality and Generalization in Emergent Languages

Compositional GAN: Learning Conditional Image Composition

Exploring Compositional Visual Generation with Latent Classifier Guidance

Compositional GAN: Learning Image-Conditional Binary Composition

Training Data Protection with Compositional Diffusion Models

Diffusion Beats Autoregressive: An Evaluation of Compositional Generation in Text-to-Image Models

Improving Compositional Generalization Using Iterated Learning and Simplicial Embeddings

Generating Intermediate Representations for Compositional Text-To-Image Generation

ControlCom: Controllable Image Composition using Diffusion Model

What makes Models Compositional? A Theoretical View: With Supplement

Rare-to-Frequent: Unlocking Compositional Generation Power of Diffusion Models on Rare Concepts with LLM Guidance