Abstract:Humans leverage compositionality to efficiently learn new concepts, understanding how familiar parts can combine together to form novel objects. In contrast, popular computer vision models struggle to make the same types of inferences, requiring more data and generalizing less flexibly than people do. Here, we study these distinctively human abilities across a range of different types of visual composition, examining how people classify and generate ``alien figures'' with rich relational structure. We also develop a Bayesian program induction model which searches for the best programs for generating the candidate visual figures, utilizing a large program space containing different compositional mechanisms and abstractions. In few shot classification tasks, we find that people and the program induction model can make a range of meaningful compositional generalizations, with the model providing a strong account of the experimental data as well as interpretable parameters that reveal human assumptions about the factors invariant to category membership (here, to rotation and changing part attachment). In few shot generation tasks, both people and the models are able to construct compelling novel examples, with people behaving in additional structured ways beyond the model capabilities, e.g. making choices that complete a set or reconfiguring existing parts in highly novel ways. To capture these additional behavioral patterns, we develop an alternative model based on neuro-symbolic program induction: this model also composes new concepts from existing parts yet, distinctively, it utilizes neural network modules to successfully capture residual statistical structure. Together, our behavioral and computational findings show how people and models can produce a rich variety of compositional behavior when classifying and generating visual objects.

Learning to Predict Novel Noun-Noun Compounds

Automatic Acquisition of Chinese Novel Noun Compounds

Transfer and Multi-Task Learning for Noun-Noun Compound Interpretation

Visualizing and Understanding Neural Models in NLP

From chocolate bunny to chocolate crocodile: Do Language Models Understand Noun Compounds?

A Novel Model for Recognition of Compounding Nouns in English and Chinese

What Happens Next? Event Prediction Using a Compositional Neural Network Model

A Systematic Comparison of English Noun Compound Representations

Learning to Infer Unseen Single-/ Multi-Attribute-Object Compositions with Graph Networks.

How is a “Kitchen Chair” like a “Farm Horse”? Exploring the Representation of Noun-Noun Compound Semantics in Transformer-based Language Models

Do Vision-Language Models Understand Compound Nouns?

A Study of Compositional Generalization in Neural Models

Using the Web as an Implicit Training Set: Application to Noun Compound Syntax and Semantics

Noun Compound Interpretation With Relation Classification and Paraphrasing

Nominal Compound Chain Extraction: A New Task for Semantic-enriched Lexical Chain

Adaptive Semantic Compositionality for Sentence Modelling.

Visualizing the Obvious: A Concreteness-based Ensemble Model for Noun Property Prediction

Compositional diversity in visual concept learning

Explicitly Representing Syntax Improves Sentence-to-layout Prediction of Unexpected Situations

Language learning and innateness: some implications of Compounds Research