Abstract:Multimodal foundation models that can holistically process text alongside images, video, audio, and other sensory modalities are increasingly used in a variety of real-world applications. However, it is challenging to characterize and study progress in multimodal foundation models, given the range of possible modeling decisions, tasks, and domains. In this paper, we introduce Holistic Evaluation of Multimodal Models (HEMM) to systematically evaluate the capabilities of multimodal foundation models across a set of 3 dimensions: basic skills, information flow, and real-world use cases. Basic multimodal skills are internal abilities required to solve problems, such as learning interactions across modalities, fine-grained alignment, multi-step reasoning, and the ability to handle external knowledge. Information flow studies how multimodal content changes during a task through querying, translation, editing, and fusion. Use cases span domain-specific challenges introduced in real-world multimedia, affective computing, natural sciences, healthcare, and human-computer interaction applications. Through comprehensive experiments across the 30 tasks in HEMM, we (1) identify key dataset dimensions (e.g., basic skills, information flows, and use cases) that pose challenges to today's models, and (2) distill performance trends regarding how different modeling dimensions (e.g., scale, pre-training data, multimodal alignment, pre-training, and instruction tuning objectives) influence performance. Our conclusions regarding challenging multimodal interactions, use cases, and tasks requiring reasoning and external knowledge, the benefits of data and model scale, and the impacts of instruction tuning yield actionable insights for future work in multimodal foundation models.

Representing and Using Knowledge with the Contextual Evaluation Model

Semantic Context Enrichment and Shifting Based on Multi-dimensional Association

A Context Quality Model to Support Transparent Reasoning with Uncertain Context

Contextualized Evaluations: Taking the Guesswork Out of Language Model Evaluations

Learning and expertise with scientific external representations: an embodied and extended cognition model

HEMM: Holistic Evaluation of Multimodal Foundation Models

MiCEval: Unveiling Multimodal Chain of Thought's Quality via Image Description and Reasoning Steps

On the Computational Modeling of Meaning: Embodied Cognition Intertwined with Emotion

CEM: Commonsense-aware Empathetic Response Generation

Evaluating the Representational Hub of Language and Vision Models

Understanding ME? Multimodal Evaluation for Fine-grained Visual Commonsense

CogExplore: Contextual Exploration with Language-Encoded Environment Representations

Emotion Recognition under Consideration of the Emotion Component Process Model

Explore until Confident: Efficient Exploration for Embodied Question Answering

Modeling human activity comprehension at human scale: prediction, segmentation, and categorization

Between Sense and Sensibility: Declarative narrativisation of mental models as a basis and benchmark for visuo-spatial cognition and computation focussed collaborative cognitive systems

EmET: Emotion Elicitation and Emotion Transition Model

Inspecting and Editing Knowledge Representations in Language Models

Evaluation of a computerized aid for creating human behavioral representations of human-computer interaction

CAT: A Contextualized Conceptualization and Instantiation Framework for Commonsense Reasoning

ECR-Chain: Advancing Generative Language Models to Better Emotion-Cause Reasoners through Reasoning Chains