Abstract:In recent times, Vision-Language Models (VLMs) have been trained under two predominant paradigms. Generative training has enabled Multimodal Large Language Models (MLLMs) to tackle various complex tasks, yet issues such as hallucinations and weak object discrimination persist. Discriminative training, exemplified by models like CLIP, excels in zero-shot image-text classification and retrieval, yet struggles with complex scenarios requiring fine-grained semantic differentiation. This paper addresses these challenges by proposing a unified approach that integrates the strengths of both paradigms. Considering interleaved image-text sequences as the general format of input samples, we introduce a structure-induced training strategy that imposes semantic relationships between input samples and the MLLM's hidden state. This approach enhances the MLLM's ability to capture global semantics and distinguish fine-grained semantics. By leveraging dynamic sequence alignment within the Dynamic Time Warping framework and integrating a novel kernel for fine-grained semantic differentiation, our method effectively balances generative and discriminative tasks. Extensive experiments demonstrate the effectiveness of our approach, achieving state-of-the-art results in multiple generative tasks, especially those requiring cognitive and discrimination abilities. Additionally, our method surpasses discriminative benchmarks in interleaved and fine-grained retrieval tasks. By employing a retrieval-augmented generation strategy, our approach further enhances performance in some generative tasks within one model, offering a promising direction for future research in vision-language modeling.

Saliency-based Multi-View Mixed Language Training for Zero-shot Cross-lingual Classification.

FC-MTLF: A Fine- and Coarse-grained Multi-Task Learning Framework for Cross-Lingual Spoken Language Understanding.

UC2: Universal Cross-lingual Cross-modal Vision-and-Language Pre-training

Multi-Lingual Acquisition on Multimodal Pre-training for Cross-modal Retrieval

Attention-Informed Mixed-Language Training for Zero-Shot Cross-Lingual Task-Oriented Dialogue Systems

Multimodal Pretraining from Monolingual to Multilingual

Cross-View Language Modeling: Towards Unified Cross-Lingual Cross-Modal Pre-training

Multilingual Multimodal Learning with Machine Translated Text

Multi-level Contrastive Learning for Cross-lingual Spoken Language Understanding

Unified Generative and Discriminative Training for Multi-modal Large Language Models

Improving Spoken Language Understanding with Cross-Modal Contrastive Learning

Learning Disentangled Semantic Representations for Zero-Shot Cross-Lingual Transfer in Multilingual Machine Reading Comprehension

Language-Driven Cross-Modal Classifier for Zero-Shot Multi-Label Image Recognition

MULTI-LEVEL CONTRASTIVE LEARNING FOR CROSS-LINGUAL ALIGNMENT

InfMLLM: A Unified Framework for Visual-Language Tasks.

Multi-label Masked Language Modeling on Zero-shot Code-switched Sentiment Analysis

mSLAM: Massively multilingual joint pre-training for speech and text

Towards Zero-shot Cross-lingual Image Retrieval and Tagging

Discrete Multimodal Transformers with a Pretrained Large Language Model for Mixed-Supervision Speech Processing

Generalizing Multimodal Pre-training into Multilingual via Language Acquisition

Zero-Shot Cross-Lingual Knowledge Transfer in VQA Via Multimodal Distillation