Abstract:Despite significant advancements, large multimodal models (LMMs) still struggle to bridge the gap between low-level visual perception -- focusing on shapes, sizes, and layouts -- and high-level language reasoning, such as semantics and logic. This limitation is evident in tasks that require precise visual perception, like comparing geometric properties or solving visual reasoning problems. To study this failure mode, we focus on vector graphics -- images composed of 2D objects and shapes, prevalent in LMM-based tasks in web, design, and OS environments. We identify two key research questions: how can we enable precise visual perception, and how can we facilitate high-level reasoning based on such low-level perceptions? To capture fine visual details, we use Scalable Vector Graphics (SVG) for accurate encoding of visual scenes. However, SVGs are not readily interpretable by LMMs in a zero-shot manner. To tackle this, we propose the Visually Descriptive Language Model (VDLM), which introduces a Primal Visual Description (PVD) as an intermediate textual representation. PVD translates SVGs into a text-based abstraction consisting of primitive attributes (e.g., shape, position, measurement) and their corresponding values. PVD can be learned using task-agnostic synthesized data and represents visual primitives that are universal across vector graphics. This abstraction is more structured, allowing for direct interpretation by foundation models for zero-shot generalization. Without human-annotated data, empirical results show that VDLM significantly improves state-of-the-art LMMs like GPT-4o on various multimodal perception and reasoning tasks. Extensive analyses of VDLM show improved interpretability due to its disentangled perception and reasoning. We also demonstrate a positive correlation between PVD quality and task performance. Project page: <a class="link-external link-https" href="https://mikewangwzhl.github.io/VDLM/" rel="external noopener nofollow">this https URL</a>

Language Models as Knowledge Bases for Visual Word Sense Disambiguation

Large Language Models and Multimodal Retrieval for Visual Word Sense Disambiguation

Declarative Knowledge Distillation from Large Language Models for Visual Question Answering Datasets

Can LLMs assist with Ambiguity? A Quantitative Evaluation of various Large Language Models on Word Sense Disambiguation

Cognitive Visual-Language Mapper: Advancing Multimodal Comprehension with Enhanced Visual Knowledge Alignment

Visually-Augmented Language Modeling

From Images to Textual Prompts: Zero-Shot Visual Question Answering with Frozen Large Language Models

Right this way: Can VLMs Guide Us to See More to Answer Questions?

ARPA: A Novel Hybrid Model for Advancing Visual Word Disambiguation Using Large Language Models and Transformers

Filling the Image Information Gap for VQA: Prompting Large Language Models to Proactively Ask Questions

Probing Visual Language Priors in VLMs

VLIS: Unimodal Language Models Guide Multimodal Language Generation

Vision Meets Definitions: Unsupervised Visual Word Sense Disambiguation Incorporating Gloss Information

Improving Zero-shot Visual Question Answering via Large Language Models with Reasoning Question Prompts

Advancing Large Multi-modal Models with Explicit Chain-of-Reasoning and Visual Question Generation

Visually Descriptive Language Model for Vector Graphics Reasoning

The Contribution of Knowledge in Visiolinguistic Learning: A Survey on Tasks and Challenges

Language Guided Visual Question Answering: Elevate Your Multimodal Language Model Using Knowledge-Enriched Prompts

Enhancing Advanced Visual Reasoning Ability of Large Language Models

Are VLMs Really Blind

Translate to Disambiguate: Zero-shot Multilingual Word Sense Disambiguation with Pretrained Language Models