Abstract:Despite significant advancements, large multimodal models (LMMs) still struggle to bridge the gap between low-level visual perception -- focusing on shapes, sizes, and layouts -- and high-level language reasoning, such as semantics and logic. This limitation is evident in tasks that require precise visual perception, like comparing geometric properties or solving visual reasoning problems. To study this failure mode, we focus on vector graphics -- images composed of 2D objects and shapes, prevalent in LMM-based tasks in web, design, and OS environments. We identify two key research questions: how can we enable precise visual perception, and how can we facilitate high-level reasoning based on such low-level perceptions? To capture fine visual details, we use Scalable Vector Graphics (SVG) for accurate encoding of visual scenes. However, SVGs are not readily interpretable by LMMs in a zero-shot manner. To tackle this, we propose the Visually Descriptive Language Model (VDLM), which introduces a Primal Visual Description (PVD) as an intermediate textual representation. PVD translates SVGs into a text-based abstraction consisting of primitive attributes (e.g., shape, position, measurement) and their corresponding values. PVD can be learned using task-agnostic synthesized data and represents visual primitives that are universal across vector graphics. This abstraction is more structured, allowing for direct interpretation by foundation models for zero-shot generalization. Without human-annotated data, empirical results show that VDLM significantly improves state-of-the-art LMMs like GPT-4o on various multimodal perception and reasoning tasks. Extensive analyses of VDLM show improved interpretability due to its disentangled perception and reasoning. We also demonstrate a positive correlation between PVD quality and task performance. Project page: <a class="link-external link-https" href="https://mikewangwzhl.github.io/VDLM/" rel="external noopener nofollow">this https URL</a>

ML 2 MG-VLCR: A Multimodal LLM Guided Zero-shot Method for Visio-linguistic Compositional Reasoning with Autoregressive Generative Language Model

Compositional Substitutivity of Visual Reasoning for Visual Question Answering

In-Context Compositional Generalization for Large Vision-Language Models

Compositional Zero-shot Learning Via Progressive Language-based Observations

Towards Truly Zero-shot Compositional Visual Reasoning with LLMs as Programmers

Enhancing Advanced Visual Reasoning Ability of Large Language Models

Interpretable Composition Attribution Enhancement for Visio-linguistic Compositional Understanding

CoVLM: Composing Visual Entities and Relationships in Large Language Models Via Communicative Decoding

Improving Context Understanding in Multimodal Large Language Models Via Multimodal Composition Learning

Causal Graphical Models for Vision-Language Compositional Understanding

Contrasting Intra-Modal and Ranking Cross-Modal Hard Negatives to Enhance Visio-Linguistic Compositional Understanding

Prompting Large Vision-Language Models for Compositional Reasoning

Vision-by-Language for Training-Free Compositional Image Retrieval

Visually Descriptive Language Model for Vector Graphics Reasoning

Zero-shot Referring Expression Comprehension via Structural Similarity Between Images and Captions

MMCOMPOSITION: Revisiting the Compositionality of Pre-trained Vision-Language Models

Insight-V: Exploring Long-Chain Visual Reasoning with Multimodal Large Language Models

Natural Language Inference Improves Compositionality in Vision-Language Models

Compositional Chain-of-Thought Prompting for Large Multimodal Models

A Surprising Failure? Multimodal LLMs and the NLVR Challenge

Multi-modal Auto-regressive Modeling via Visual Words