Abstract:The advent of the Transformer has led to the development of large language models (LLM), which appear to demonstrate human-like capabilities. To assess the generality of this class of models and a variety of other base neural network architectures to multimodal domains, we evaluated and compared their capacity for multimodal generalization. We introduce a multimodal question-answer benchmark to evaluate three specific types of out-of-distribution (OOD) generalization performance: distractor generalization (generalization in the presence of distractors), systematic compositional generalization (generalization to new task permutations), and productive compositional generalization (generalization to more complex tasks structures). We found that across model architectures (e.g., RNNs, Transformers, Perceivers, etc.), models with multiple attention layers, or models that leveraged cross-attention mechanisms between input domains, fared better. Our positive results demonstrate that for multimodal distractor and systematic generalization, either cross-modal attention or models with deeper attention layers are key architectural features required to integrate multimodal inputs. On the other hand, neither of these architectural features led to productive generalization, suggesting fundamental limitations of existing architectures for specific types of multimodal generalization. These results demonstrate the strengths and limitations of specific architectural components underlying modern neural models for multimodal reasoning. Finally, we provide Generic COG (gCOG), a configurable benchmark with several multimodal generalization splits, for future studies to explore.

How Do We Generalize

Generalization in Generative Adversarial Networks: A Novel Perspective from Privacy Protection.

Unifying Principles of Generalization: Past, Present, and Future

Generalization in Cooperative Multi-Agent Systems

Generalization in Neural Networks: A Broad Survey

Aligning Generalisation Between Humans and Machines

A General Theory for Compositional Generalization

Improving Systematic Generalization Through Modularity and Augmentation

A neural mechanism for compositional generalization of structure in humans

Generative Adversarial Collaborations: A practical guide for conference organizers and participating scientists

Representations and generalization in artificial and brain neural networks

A roadmap to reverse engineering real-world generalization by combining naturalistic paradigms, deep sampling, and predictive computational models

Position Paper: Generalized grammar rules and structure-based generalization beyond classical equivariance for lexical tasks and transduction

On Computation and Generalization of Generative Adversarial Imitation Learning.

On the generalization capacity of neural networks during generic multimodal reasoning

Compositional generalization through abstract representations in human and artificial neural networks

A Benchmark for Systematic Generalization in Grounded Language Understanding

Projective simulation with generalization

On Generalization for Generative Flow Networks

On the Anomalous Generalization of GANs

Building Object-based Causal Programs for Human-like Generalization