Abstract:Large Language Models (LLMs) have made the ambitious quest for generalist agents significantly far from being a fantasy. A key hurdle for building such general models is the diversity and heterogeneity of tasks and modalities. A promising solution is unification, allowing the support of a myriad of tasks and modalities within one unified framework. While few large models (e.g., Flamingo (Alayrac et al., 2022), trained on massive datasets, can support more than two modalities, current small to mid-scale unified models are still limited to 2 modalities, usually image-text or video-text. The question that we ask is: is it possible to build efficiently a unified model that can support all modalities? To answer this, we propose UnIVAL, a step further towards this ambitious goal. Without relying on fancy datasets sizes or models with billions of parameters, the ~ 0.25B parameter UnIVAL model goes beyond two modalities and unifies text, images, video, and audio into a single model. Our model is efficiently pretrained on many tasks, based on task balancing and multimodal curriculum learning. UnIVAL shows competitive performance to existing state-of-the-art approaches, across image and video-text tasks. The feature representations learned from image and video-text modalities, allows the model to achieve competitive performance when finetuned on audio-text tasks, despite not being pretrained on audio. Thanks to the unified model, we propose a novel study on multimodal model merging via weight interpolation of models trained on different multimodal tasks, showing their benefits in particular for out-of-distribution generalization. Finally, we motivate unification by showing the synergy between tasks. The model weights and code are released here: <a class="link-external link-https" href="https://github.com/mshukor/UnIVAL" rel="external noopener nofollow">this https URL</a>.

Split to Merge: Unifying Separated Modalities for Unsupervised Domain Adaptation

Multi-Modal Adapter for Vision-Language Models

UMFC: Unsupervised Multi-Domain Feature Calibration for Vision-Language Models

VLP2MSA: Expanding Vision-Language Pre-Training to Multimodal Sentiment Analysis

Unified Generative and Discriminative Training for Multi-modal Large Language Models

UNIMO-2: End-to-End Unified Vision-Language Grounded Learning

UNIMO: Towards Unified-Modal Understanding and Generation Via Cross-Modal Contrastive Learning

Uni-MoE: Scaling Unified Multimodal LLMs with Mixture of Experts

Unifying Cross-Lingual and Cross-Modal Modeling Towards Weakly Supervised Multilingual Vision-Language Pre-training

VLMixer: Unpaired Vision-Language Pre-training via Cross-Modal CutMix

From Unimodal to Multimodal: Scaling up Projectors to Align Modalities

Multi-Modal Video Topic Segmentation with Dual-Contrastive Domain Adaptation

Cluster, Split, Fuse, and Update: Meta-Learning for Open Compound Domain Adaptive Semantic Segmentation

Demonstrative Instruction Following in Multimodal LLMs Via Integrating Low-Rank Adaptation with Ensemble Learning

OSAN: A One-Stage Alignment Network to Unify Multimodal Alignment and Unsupervised Domain Adaptation

Unicoder-VL: A Universal Encoder for Vision and Language by Cross-modal Pre-training

Multimodal Representation Learning by Alternating Unimodal Adaptation

UniDiff: Advancing Vision-Language Models with Generative and Discriminative Learning

Improving Multi-Modal Learning with Uni-Modal Teachers

UnIVAL: Unified Model for Image, Video, Audio and Language Tasks

MoVA: Adapting Mixture of Vision Experts to Multimodal Context