Abstract:While visual language model architectures and training infrastructures advance rapidly, data curation remains under-explored where quantity and quality become a bottleneck. Existing work either crawls extra Internet data with a loose guarantee of quality or distills from black-box proprietary models, e.g., GPT-4V / Gemini that are API frequency and performance bounded. This work enables a VLM to improve itself via data enhancement, exploiting its generative nature. We introduce a simple yet effective VLM augmentation scheme that includes a self-augment step and a specialist-augment step to iteratively improve data quality and hence, model performance. In the self-augment step, the instruction-finetuned VLM recaptions its pretraining caption datasets and then retrains from scratch leveraging refined data. Without any expensive human-in-the-loop annotation, we observe improvements in data quality and downstream accuracy boosts with three self-augmentation rounds -- a viable free lunch to the current VLM training recipe. When self-augmentation saturates, we augment the caption diversity by leveraging specialty skills picked up from instruction finetuning. We finetune VLM specialists from the self-augmented VLM with domain-specific experts, including spatial, grounding, and OCR, to fuse task-aware synthetic data into the pretraining stage. Data quality improvements and hallucination reductions are cross-checked by VLM (GPT-4V, Gemini) and human judges. Combining self-augmentation and specialist-augmented training, VILA$^2$ consistently improves the accuracy on a wide range of benchmarks over the prior art, producing a reusable pretraining dataset that is 300x more cost-efficient than human labeling.

Bag of Tricks for Effective Language Model Pretraining and Downstream Adaptation: A Case Study on GLUE

Toward Efficient Language Model Pretraining and Downstream Adaptation via Self-Evolution: A Case Study on SuperGLUE

GRILL: Grounded Vision-language Pre-training via Aligning Text and Image Regions

GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding

Behind the Scene: Revealing the Secrets of Pre-trained Vision-and-Language Models

VindLU: A Recipe for Effective Video-and-Language Pretraining

SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systems

GLGE: A New General Language Generation Evaluation Benchmark

VECO: Variable and Flexible Cross-lingual Pre-training for Language Understanding and Generation

Pre-Training a Language Model Without Human Language

Enhancing CLIP with GPT-4: Harnessing Visual Descriptions as Prompts

Improving the Language Understanding Capabilities of Large Language Models Using Reinforcement Learning

How to Determine the Most Powerful Pre-trained Language Model Without Brute Force Fine-tuning? an Empirical Survey

GLUE-X: Evaluating Natural Language Understanding Models from an Out-of-Distribution Generalization Perspective.

Exploring the Impact of a Transformer's Latent Space Geometry on Downstream Task Performance

VideoGLUE: Video General Understanding Evaluation of Foundation Models

VILA$^2$: VILA Augmented VILA

Empirical Analysis of Efficient Fine-Tuning Methods for Large Pre-Trained Language Models

GRIT-VLP: Grouped Mini-batch Sampling for Efficient Vision and Language Pre-training

Rethinking Overlooked Aspects in Vision-Language Models

IdealGPT: Iteratively Decomposing Vision and Language Reasoning via Large Language Models