Abstract:Existing vision-language models (VLMs) mostly rely on vision encoders to extract visual features followed by large language models (LLMs) for visual-language tasks. However, the vision encoders set a strong inductive bias in abstracting visual representation, e.g., resolution, aspect ratio, and semantic priors, which could impede the flexibility and efficiency of the VLMs. Training pure VLMs that accept the seamless vision and language inputs, i.e., without vision encoders, remains challenging and rarely explored. Empirical observations reveal that direct training without encoders results in slow convergence and large performance gaps. In this work, we bridge the gap between encoder-based and encoder-free models, and present a simple yet effective training recipe towards pure VLMs. Specifically, we unveil the key aspects of training encoder-free VLMs efficiently via thorough experiments: (1) Bridging vision-language representation inside one unified decoder; (2) Enhancing visual recognition capability via extra supervision. With these strategies, we launch EVE, an encoder-free vision-language model that can be trained and forwarded efficiently. Notably, solely utilizing 35M publicly accessible data, EVE can impressively rival the encoder-based VLMs of similar capacities across multiple vision-language benchmarks. It significantly outperforms the counterpart Fuyu-8B with mysterious training procedures and undisclosed training data. We believe that EVE provides a transparent and efficient route for developing a pure decoder-only architecture across modalities. Our code and models are publicly available at: <a class="link-external link-https" href="https://github.com/baaivision/EVE" rel="external noopener nofollow">this https URL</a>.

Vocabulary-informed Language Encoding.

A Multimodal Visual Encoding Model Aided by Introducing Verbal Semantic Information

Revisiting Language Encoding in Learning Multilingual Representations

Embedding structure matters: Comparing methods to adapt multilingual vocabularies to new languages

Language-Informed Visual Concept Learning

Beyond Bilingual: Multi-sense Word Embeddings using Multilingual Context

Improving Multilingual Models with Language-Clustered Vocabularies

Language Models are Universal Embedders

MYTE: Morphology-Driven Byte Encoding for Better and Fairer Multilingual Language Modeling

A Variational Autoencoding Approach for Inducing Cross-lingual Word Embeddings

Learning Multilingual Topics with Neural Variational Inference

Vision as an Interlingua: Learning Multilingual Semantic Embeddings of Untranscribed Speech

Informative Language Representation Learning for Massively Multilingual Neural Machine Translation

Unifying Visual-Semantic Embeddings with Multimodal Neural Language Models

Modular Sentence Encoders: Separating Language Specialization from Cross-Lingual Alignment

Investigating Language Universal and Specific Properties in Word Embeddings

How Vocabulary Sharing Facilitates Multilingualism in LLaMA?

Unveiling Encoder-Free Vision-Language Models

Embedding Word Similarity with Neural Machine Translation

Brain encoding models based on multimodal transformers can transfer across language and vision

Empowering MultiModal Models' In-Context Learning Ability through Large Language Models.