Abstract:In recent times, Vision-Language Models (VLMs) have been trained under two predominant paradigms. Generative training has enabled Multimodal Large Language Models (MLLMs) to tackle various complex tasks, yet issues such as hallucinations and weak object discrimination persist. Discriminative training, exemplified by models like CLIP, excels in zero-shot image-text classification and retrieval, yet struggles with complex scenarios requiring fine-grained semantic differentiation. This paper addresses these challenges by proposing a unified approach that integrates the strengths of both paradigms. Considering interleaved image-text sequences as the general format of input samples, we introduce a structure-induced training strategy that imposes semantic relationships between input samples and the MLLM's hidden state. This approach enhances the MLLM's ability to capture global semantics and distinguish fine-grained semantics. By leveraging dynamic sequence alignment within the Dynamic Time Warping framework and integrating a novel kernel for fine-grained semantic differentiation, our method effectively balances generative and discriminative tasks. Extensive experiments demonstrate the effectiveness of our approach, achieving state-of-the-art results in multiple generative tasks, especially those requiring cognitive and discrimination abilities. Additionally, our method surpasses discriminative benchmarks in interleaved and fine-grained retrieval tasks. By employing a retrieval-augmented generation strategy, our approach further enhances performance in some generative tasks within one model, offering a promising direction for future research in vision-language modeling.

A novel framework of training hidden markov support vector machines from lightly-annotated data.

Semi-supervised Learning of Statistical Models for Natural Language Understanding.

Hierarchical and Bidirectional Joint Multi-Task Classifiers for Natural Language Understanding

Learning Conditional Random Fields From Unaligned Data For Natural Language Understanding

A Hybrid Generative/Discriminative Framework to Train a Semantic Parser from an Un-annotated Corpus.

Discriminative Training of the Hidden Vector State Model for Semantic Parsing

Learning Latent Semantic Annotations for Grounding Natural Language to Structured Data

Training the Hidden Vector State Model from Un-annotated Corpus

Spoken Language Understanding from Unaligned Data Using Discriminative Classification Models

A Joint Learning Framework with BERT for Spoken Language Understanding.

A Bi-model based RNN Semantic Frame Parsing Model for Intent Detection and Slot Filling

Dual Learning for Semi-Supervised Natural Language Understanding

Unified Generative and Discriminative Training for Multi-modal Large Language Models

A New Pre-Training Method for Training Deep Learning Models with Application to Spoken Language Understanding

Joint Intent Detection Model for Task-oriented Human-Computer Dialogue System Using Asynchronous Training

Tomato, Tomahto, Tomate: Measuring the Role of Shared Semantics among Subwords in Multilingual Language Models

A Joint Multi-Task Learning Framework for Spoken Language Understanding

InfMLLM: A Unified Framework for Visual-Language Tasks.

Joint Spoken Language Understanding And Domain Adaptive Language Modeling

Dense Multimodal Alignment for Open-Vocabulary 3D Scene Understanding

Understanding Long Videos with Multimodal Language Models