Abstract:How well are unimodal vision and language models aligned? Although prior work have approached answering this question, their assessment methods do not directly translate to how these models are used in practical vision-language tasks. In this paper, we propose a direct assessment method, inspired by linear probing, to assess vision-language alignment. We identify that the degree of alignment of the SSL vision models depends on their SSL training objective, and we find that the clustering quality of SSL representations has a stronger impact on alignment performance than their linear separability. Next, we introduce Swift Alignment of Image and Language (SAIL), a efficient transfer learning framework that aligns pretrained unimodal vision and language models for downstream vision-language tasks. Since SAIL leverages the strengths of pretrained unimodal models, it requires significantly fewer (6%) paired image-text data for the multimodal alignment compared to models like CLIP which are trained from scratch. SAIL training only requires a single A100 GPU, 5 hours of training and can accommodate a batch size up to 32,768. SAIL achieves 73.4% zero-shot accuracy on ImageNet (vs. CLIP's 72.7%) and excels in zero-shot retrieval, complex reasoning, and semantic segmentation. Additionally, SAIL improves the language-compatibility of vision encoders that in turn enhance the performance of multimodal large language models. The entire codebase and model weights are open-source: <a class="link-external link-https" href="https://lezhang7.github.io/sail.github.io/" rel="external noopener nofollow">this https URL</a>

Self-Augmented In-Context Learning for Unsupervised Word Translation

Effective Self-Mining of In-Context Examples for Unsupervised Machine Translation with LLMs

On Bilingual Lexicon Induction with Large Language Models

On Task Performance and Model Calibration with Supervised and Self-Ensembled In-Context Learning

DAIL: Data Augmentation for In-Context Learning via Self-Paraphrase

Improving In-Context Learning with Small Language Model Ensembles

SAIL: Search-Augmented Instruction Learning

Self-Augmentation Improves Zero-Shot Cross-Lingual Transfer

Towards Multimodal In-Context Learning for Vision & Language Models

Language Model Self-improvement by Reinforcement Learning Contemplation

Improving Word Translation via Two-Stage Contrastive Learning

Analyzing and Adapting Large Language Models for Few-Shot Multilingual NLU: Are We There Yet?

Self-Translate-Train: Enhancing Cross-Lingual Transfer of Large Language Models via Inherent Capability

In-Context Learning with Reinforcement Learning for Incomplete Utterance Rewriting

SSP: Self-Supervised Prompting for Cross-Lingual Transfer to Low-Resource Languages using Large Language Models

Investigating the Learning Behaviour of In-Context Learning: A Comparison with Supervised Learning

Assessing and Learning Alignment of Unimodal Vision and Language Models

Decoding In-Context Learning: Neuroscience-inspired Analysis of Representations in Large Language Models

LLMs Are In-Context Reinforcement Learners

Does In-Context Learning Really Learn? Rethinking How Large Language Models Respond and Solve Tasks via In-Context Learning

An Empirical Study of In-context Learning in LLMs for Machine Translation