Abstract:Despite significant advancements, large multimodal models (LMMs) still struggle to bridge the gap between low-level visual perception -- focusing on shapes, sizes, and layouts -- and high-level language reasoning, such as semantics and logic. This limitation is evident in tasks that require precise visual perception, like comparing geometric properties or solving visual reasoning problems. To study this failure mode, we focus on vector graphics -- images composed of 2D objects and shapes, prevalent in LMM-based tasks in web, design, and OS environments. We identify two key research questions: how can we enable precise visual perception, and how can we facilitate high-level reasoning based on such low-level perceptions? To capture fine visual details, we use Scalable Vector Graphics (SVG) for accurate encoding of visual scenes. However, SVGs are not readily interpretable by LMMs in a zero-shot manner. To tackle this, we propose the Visually Descriptive Language Model (VDLM), which introduces a Primal Visual Description (PVD) as an intermediate textual representation. PVD translates SVGs into a text-based abstraction consisting of primitive attributes (e.g., shape, position, measurement) and their corresponding values. PVD can be learned using task-agnostic synthesized data and represents visual primitives that are universal across vector graphics. This abstraction is more structured, allowing for direct interpretation by foundation models for zero-shot generalization. Without human-annotated data, empirical results show that VDLM significantly improves state-of-the-art LMMs like GPT-4o on various multimodal perception and reasoning tasks. Extensive analyses of VDLM show improved interpretability due to its disentangled perception and reasoning. We also demonstrate a positive correlation between PVD quality and task performance. Project page: <a class="link-external link-https" href="https://mikewangwzhl.github.io/VDLM/" rel="external noopener nofollow">this https URL</a>

VisB: A Lightweight Tool to Visualize Formal Models with SVG Graphics

Generating interactive documents for domain-specific validation of formal models

Grounded Language Design for Lightweight Diagramming for Formal Methods

Visually Descriptive Language Model for Vector Graphics Reasoning

VisTML: A Visual Modeling Language for Model Transformation

VisAhoi: Towards a Library to Generate and Integrate Visualization Onboarding Using High-level Visualization Grammars

SVG web environment for z specification language

A Graphical Modeling Language for Model Transformations

EVM: Incorporating Model Checking into Exploratory Visual Analysis

VMC: A Grammar for Visualizing Statistical Model Checks

Visualization by Example

VisAct: a Visualization Design System Based on Semantic Actions

ViSpec: A graphical tool for elicitation of MTL requirements

paraSBOLv: a foundation for standard-compliant genetic design visualization tools

VGBench: Evaluating Large Language Models on Vector Graphics Understanding and Generation

Stakeholder Engagement Visualization (EnVision) System for SysML Diagram Exploration

Beyond Pixels: Exploring Human-Readable SVG Generation for Simple Images with Vision Language Models

VisAnatomy: An SVG Chart Corpus with Fine-Grained Semantic Labels

A Visual Analytics Environment for Navigating Large Conceptual Models by Leveraging Generative Artificial Intelligence

ModelViz: A Model-Driven Engineering Approach for Visual Analytics System Design

Visually Representing the Landscape of Mathematical Structures