Abstract:Recently, the remarkable advance of the Large Language Model (LLM) has inspired researchers to transfer its extraordinary reasoning capability to both vision and language data. However, the prevailing approaches primarily regard the visual input as a prompt and focus exclusively on optimizing the text generation process conditioned upon vision content by a frozen LLM. Such an inequitable treatment of vision and language heavily constrains the model's potential. In this paper, we break through this limitation by representing both vision and language in a unified form. Specifically, we introduce a well-designed visual tokenizer to translate the non-linguistic image into a sequence of discrete tokens like a foreign language that LLM can read. The resulting visual tokens encompass high-level semantics worthy of a word and also support dynamic sequence length varying from the image. Coped with this tokenizer, the presented foundation model called LaVIT can handle both image and text indiscriminately under the same generative learning paradigm. This unification empowers LaVIT to serve as an impressive generalist interface to understand and generate multi-modal content simultaneously. Extensive experiments further showcase that it outperforms the existing models by a large margin on massive vision-language tasks. Our code and models are available at <a class="link-external link-https" href="https://github.com/jy0205/LaVIT" rel="external noopener nofollow">this https URL</a>.

What problem does this paper attempt to address?

The paper aims to address the issue of unequal treatment in Multimodal Large Language Models (MLLMs) when processing image and text data, and attempts to enhance the model's multimodal understanding and generation capabilities through a unified representation approach. Specifically, the paper proposes a new model named LaVIT (Language-Vision Transformer), whose main objectives include: 1. **Unified Visual and Language Representation**: Current mainstream methods typically treat visual input as prompts, optimizing only the process of generating text based on visual content. This unbalanced approach limits the model's potential. LaVIT achieves equal treatment of images and text by converting images into discrete visual tokens, allowing these tokens to be understood by large language models just like text. 2. **Dynamic Discrete Visual Tokens**: To further improve representation compatibility and computational efficiency, the paper develops a mechanism capable of producing dynamic-length discrete visual tokens. This method not only reduces redundancy among visual patches but also enhances the consistency of image and text representations within the LLM. 3. **End-to-End Training Framework**: By introducing a novel visual tokenizer, LaVIT can directly predict the next image or text token without modifying the original architecture. This allows the model to handle image and text sequences simultaneously under a unified autoregressive training paradigm. In summary, LaVIT aims to overcome the unequal treatment of visual and language processing in existing methods, achieving more efficient and unified multimodal information processing capabilities through innovative visual tokenization techniques. Experimental results show that LaVIT significantly outperforms existing models on various vision-language tasks.

Unified Language-Vision Pretraining in LLM with Dynamic Discrete Visual Tokenization

Video-LaVIT: Unified Video-Language Pre-training with Decoupled Visual-Motional Tokenization

Unified Vision-Language Pre-Training for Image Captioning and VQA

Video-LLaVA: Learning United Visual Representation by Alignment Before Projection

VatLM: Visual-Audio-Text Pre-Training with Unified Masked Prediction for Speech Representation Learning

InfMLLM: A Unified Framework for Visual-Language Tasks.

Chat-UniVi: Unified Visual Representation Empowers Large Language Models with Image and Video Understanding

Unified Generative and Discriminative Training for Multi-modal Large Language Models

u-LLaVA: Unifying Multi-Modal Tasks via Large Language Model

Leveraging per Image-Token Consistency for Vision-Language Pre-training

Beyond Text: Frozen Large Language Models in Visual Signal Comprehension

Unifying Cross-Lingual and Cross-Modal Modeling Towards Weakly Supervised Multilingual Vision-Language Pre-training

AdaptVision: Dynamic Input Scaling in MLLMs for Versatile Scene Understanding

VisionLLM: Large Language Model is also an Open-Ended Decoder for Vision-Centric Tasks

Position-Enhanced Visual Instruction Tuning for Multimodal Large Language Models

Dynamic-VLM: Simple Dynamic Visual Token Compression for VideoLLM

Reformulating Vision-Language Foundation Models and Datasets Towards Universal Multimodal Assistants

Unified Multimodal Pre-training and Prompt-based Tuning for Vision-Language Understanding and Generation

Visually-Augmented Language Modeling

ViLTA: Enhancing Vision-Language Pre-training Through Textual Augmentation

Efficient Multi-modal Large Language Models via Visual Token Grouping