Abstract:Visual instruction datasets from various distributors are released at different times and often contain a significant number of semantically redundant text-image pairs, depending on their task compositions (i.e., skills) or reference sources. This redundancy greatly limits the efficient deployment of lifelong adaptable multimodal large language models, hindering their ability to refine existing skills and acquire new competencies over time. To address this, we reframe the problem of Lifelong Instruction Tuning (LiIT) via data selection, where the model automatically selects beneficial samples to learn from earlier and new datasets based on the current state of acquired knowledge in the model. Based on empirical analyses that show that selecting the best data subset using a static importance measure is often ineffective for multi-task datasets with evolving distributions, we propose Adapt-$\infty$, a new multi-way and adaptive data selection approach that dynamically balances sample efficiency and effectiveness during LiIT. We construct pseudo-skill clusters by grouping gradient-based sample vectors. Next, we select the best-performing data selector for each skill cluster from a pool of selector experts, including our newly proposed scoring function, Image Grounding score. This data selector samples a subset of the most important samples from each skill cluster for training. To prevent the continuous increase in the size of the dataset pool during LiIT, which would result in excessive computation, we further introduce a cluster-wise permanent data pruning strategy to remove the most semantically redundant samples from each cluster, keeping computational requirements manageable. Training with samples selected by Adapt-$\infty$ alleviates catastrophic forgetting, especially for rare tasks, and promotes forward transfer across the continuum using only a fraction of the original datasets.

LITMUS Predictor: An AI Assistant for Building Reliable, High-Performing and Fair Multilingual NLP Systems

Beyond Static Models and Test Sets: Benchmarking the Potential of Pre-trained Models Across Tasks and Languages

Boosting Zero-Shot Crosslingual Performance using LLM-Based Augmentations with Effective Data Selection

Bridging the Gap: Dynamic Learning Strategies for Improving Multilingual Performance in LLMs

INCLUDE: Evaluating Multilingual Language Understanding with Regional Knowledge

LIMIT: Language Identification, Misidentification, and Translation using Hierarchical Models in 350+ Languages

Understanding and Mitigating Language Confusion in LLMs

The Language Interpretability Tool: Extensible, Interactive Visualizations and Analysis for NLP Models

PARIKSHA: A Large-Scale Investigation of Human-LLM Evaluator Agreement on Multilingual and Multi-Cultural Data

Are Large Language Model-based Evaluators the Solution to Scaling Up Multilingual Evaluation?

LLM Performance Predictors are good initializers for Architecture Search

Analyzing and Adapting Large Language Models for Few-Shot Multilingual NLU: Are We There Yet?

Cross-Lingual Auto Evaluation for Assessing Multilingual LLMs

ProxyLM: Predicting Language Model Performance on Multilingual Tasks via Proxy Models

Predicting Learning Performance with Large Language Models: A Study in Adult Literacy

MEGA: Multilingual Evaluation of Generative AI

Adapt-$\infty$: Scalable Lifelong Multimodal Instruction Tuning via Dynamic Data Selection

Hints-In-Browser: Benchmarking Language Models for Programming Feedback Generation

Improving Bilingual Capabilities of Language Models to Support Diverse Linguistic Practices in Education

NatLan: Native Language Prompting Facilitates Knowledge Elicitation Through Language Trigger Provision and Domain Trigger Retention

Adaptation with Self-Evaluation to Improve Selective Prediction in LLMs