Abstract:Large Language Models (LLMs) have demonstrated impressive performance on multimodal tasks, without any multimodal finetuning. They are the building block for Large Multimodal Models, yet, we still lack a proper understanding of their success. In this work, we expose frozen LLMs to image, video, audio and text inputs and analyse their internal representation aiming to understand their generalization beyond textual inputs. Findings. Perceptual tokens (1) are easily distinguishable from textual ones inside LLMs, with significantly different representations, and complete translation to textual tokens does not exist. Yet, (2) both perceptual and textual tokens activate similar LLM weights. Despite being different, (3) perceptual and textual tokens are implicitly aligned inside LLMs, we call this the implicit multimodal alignment (IMA), and argue that this is linked to architectural design, helping LLMs to generalize. This provide more evidence to believe that the generalization of LLMs to multimodal inputs is mainly due to their architecture. Implications. (1) We find a positive correlation between the implicit alignment score and the task performance, suggesting that this could act as a proxy metric for model evaluation and selection. (2) A negative correlation exists regarding hallucinations, revealing that this problem is mainly due to misalignment between the internal perceptual and textual representations. (3) Perceptual tokens change slightly throughout the model, thus, we propose different approaches to skip computations (e.g. in FFN layers), and significantly reduce the inference cost. (4) Due to the slowly changing embeddings across layers, and the high overlap between textual and multimodal activated weights, we compress LLMs by keeping only 1 subnetwork that works well across a wide range of multimodal tasks. Paper code: <a class="link-external link-https" href="https://github.com/mshukor/ima-lmms" rel="external noopener nofollow">this https URL</a>.

F-LMM: Grounding Frozen Large Multimodal Models

Emerging Pixel Grounding in Large Multimodal Models Without Grounding Supervision

GLaMM: Pixel Grounding Large Multimodal Model

VideoGLaMM: A Large Multimodal Model for Pixel-Level Visual Grounding in Videos

GROUNDHOG: Grounding Large Language Models to Holistic Segmentation

LLM-Optic: Unveiling the Capabilities of Large Language Models for Universal Visual Grounding

Learning to Ground VLMs without Forgetting

MC-Bench: A Benchmark for Multi-Context Visual Grounding in the Era of MLLMs

Implicit Multimodal Alignment: On the Generalization of Frozen LLMs to Multimodal Inputs

LLM-Grounder: Open-Vocabulary 3D Visual Grounding with Large Language Model as an Agent

LLaVA-Grounding: Grounded Visual Chat with Large Multimodal Models

MM1.5: Methods, Analysis & Insights from Multimodal LLM Fine-tuning

Grounded-VideoLLM: Sharpening Fine-grained Temporal Grounding in Video Large Language Models

Plug-and-Play Grounding of Reasoning in Multimodal Large Language Models

Advancing Fine-Grained Visual Understanding with Multi-Scale Alignment in Multi-Modal Models

InfMLLM: A Unified Framework for Visual-Language Tasks.

See, Say, and Segment: Teaching LMMs to Overcome False Premises

LLM4VG: Large Language Models Evaluation for Video Grounding

Grounding Multimodal Large Language Models in Actions

LM4LV: A Frozen Large Language Model for Low-level Vision Tasks

Grounding Large Language Models In Embodied Environment With Imperfect World Models