Abstract:A major reason behind the recent success of large language models (LLMs) is their \textit{in-context learning} capability, which makes it possible to rapidly adapt them to downstream text-based tasks by prompting them with a small number of relevant demonstrations. While large vision-language models (VLMs) have recently been developed for tasks requiring both text and images, they largely lack in-context learning over visual information, especially in understanding and generating text about videos. In this work, we implement \textbf{E}mergent \textbf{I}n-context \textbf{Le}arning on \textbf{V}ideos (\eilev{}), a novel training paradigm that induces in-context learning over video and text by capturing key properties of pre-training data found by prior work to be essential for in-context learning in transformers. In our experiments, we show that \eilev-trained models outperform other off-the-shelf VLMs in few-shot video narration for novel, rare actions. Furthermore, we demonstrate that these key properties of bursty distributions, skewed marginal distributions, and dynamic meaning each contribute to varying degrees to VLMs' in-context learning capability in narrating procedural videos. Our results, analysis, and \eilev{}-trained models yield numerous insights about the emergence of in-context learning over video and text, creating a foundation for future work to optimize and scale VLMs for open-domain video understanding and reasoning. Our code and demo are available at \url{<a class="link-external link-https" href="https://github.com/yukw777/EILEV" rel="external noopener nofollow">this https URL</a>}.

Diluie: Constructing Diverse Demonstrations of In-Context Learning with Large Language Model for Unified Information Extraction

In-Context Compositional Generalization for Large Vision-Language Models

RUIE: Retrieval-based Unified Information Extraction using Large Language Model

Large Language Models Know What Makes Exemplary Contexts

Unraveling the Mechanics of Learning-Based Demonstration Selection for In-Context Learning

Does In-Context Learning Really Learn? Rethinking How Large Language Models Respond and Solve Tasks via In-Context Learning

ADELIE: Aligning Large Language Models on Information Extraction

Revisiting Demonstration Selection Strategies in In-Context Learning

IEPile: Unearthing Large Scale Schema-Conditioned Information Extraction Corpus

Effective Demonstration Annotation for In-Context Learning via Language Model-Based Determinantal Point Process

Eliciting In-Context Learning in Vision-Language Models for Videos Through Curated Data Distributional Properties

Compositional Exemplars for In-context Learning

The Impact of Demonstrations on Multilingual In-Context Learning: A Multidimensional Analysis

Visual In-Context Learning for Large Vision-Language Models

ICLEval: Evaluating In-Context Learning Ability of Large Language Models

Towards Understanding In-Context Learning with Contrastive Demonstrations and Saliency Maps

DRUM: Learning Demonstration Retriever for Large MUlti-modal Models

Unified Demonstration Retriever for In-Context Learning

Supervised Knowledge Makes Large Language Models Better In-context Learners

Let's Learn Step by Step: Enhancing In-Context Learning Ability with Curriculum Learning