Abstract:A remarkable ability of human beings resides in compositional reasoning, i.e., the capacity to make "infinite use of finite means". However, current large vision-language foundation models (VLMs) fall short of such compositional abilities due to their "bag-of-words" behaviors and inability to construct words that correctly represent visual entities and the relations among the entities. To this end, we propose CoVLM, which can guide the LLM to explicitly compose visual entities and relationships among the text and dynamically communicate with the vision encoder and detection network to achieve vision-language communicative decoding. Specifically, we first devise a set of novel communication tokens for the LLM, for dynamic communication between the visual detection system and the language system. A communication token is generated by the LLM following a visual entity or a relation, to inform the detection network to propose regions that are relevant to the sentence generated so far. The proposed regions-of-interests (ROIs) are then fed back into the LLM for better language generation contingent on the relevant regions. The LLM is thus able to compose the visual entities and relationships through the communication tokens. The vision-to-language and language-to-vision communication are iteratively performed until the entire sentence is generated. Our framework seamlessly bridges the gap between visual perception and LLMs and outperforms previous VLMs by a large margin on compositional reasoning benchmarks (e.g., ~20% in HICO-DET mAP, ~14% in Cola top-1 accuracy, and ~3% on ARO top-1 accuracy). We also achieve state-of-the-art performances on traditional vision-language tasks such as referring expression comprehension and visual question answering.

Combining Lattice-Based Language Dependent and Independent Approaches for Out-of-Language Detection in LVCSR.

Phone Lattice Reconstruction for Embedded Language Recognition in LVCSR

Addressing the Out-of-vocabulary Problem for Large-Scale Chinese Spoken Term Detection

Combining Chinese Spoken Term Detection Systems Via Side-Information Conditioned Linear Logistic Regression

Towards Unified Spoken Language Understanding Decoding via Label-aware Compact Linguistics Representations

LVCSR System for English and Mandarin Integrated with Language Identification

Visual Information Assisted Mandarin Large Vocabulary Continuous Speech Recognition

Integrating Lattice-Free MMI into End-to-End Speech Recognition

Fusing Multiple Systems into a Compact Lattice Index for Chinese Spoken Term Detection.

A Trellis Based Fast Lattice Generating Algorithm

Mitigating Hallucinations in Large Vision-Language Models (LVLMs) via Language-Contrastive Decoding (LCD)

Complementary Combination in I-Vector Level for Language Recognition.

Language-aware PLDA for multilingual speaker recognition

Streaming Language Identification using Combination of Acoustic Representations and ASR Hypotheses

A unified system for multilingual speech recognition and language identification

Combination Of Data Borrowing Strategies For Low-Resource Lvcsr

An Empirical Study of Language Model Integration for Transducer Based Speech Recognition

CoVLM: Composing Visual Entities and Relationships in Large Language Models Via Communicative Decoding

Lattice-based Indexing for Spontaneous Mandarin Speech

Improvement Comparison of Different Lattice-based Discriminative Training Methods in Chinese-monolingual and Chinese-English-bilingual Speech Recognition

Towards Vocabulary-Independent Speech Indexing for Large-Scale Repositories