Abstract:A remarkable ability of human beings resides in compositional reasoning, i.e., the capacity to make "infinite use of finite means". However, current large vision-language foundation models (VLMs) fall short of such compositional abilities due to their "bag-of-words" behaviors and inability to construct words that correctly represent visual entities and the relations among the entities. To this end, we propose CoVLM, which can guide the LLM to explicitly compose visual entities and relationships among the text and dynamically communicate with the vision encoder and detection network to achieve vision-language communicative decoding. Specifically, we first devise a set of novel communication tokens for the LLM, for dynamic communication between the visual detection system and the language system. A communication token is generated by the LLM following a visual entity or a relation, to inform the detection network to propose regions that are relevant to the sentence generated so far. The proposed regions-of-interests (ROIs) are then fed back into the LLM for better language generation contingent on the relevant regions. The LLM is thus able to compose the visual entities and relationships through the communication tokens. The vision-to-language and language-to-vision communication are iteratively performed until the entire sentence is generated. Our framework seamlessly bridges the gap between visual perception and LLMs and outperforms previous VLMs by a large margin on compositional reasoning benchmarks (e.g., ~20% in HICO-DET mAP, ~14% in Cola top-1 accuracy, and ~3% on ARO top-1 accuracy). We also achieve state-of-the-art performances on traditional vision-language tasks such as referring expression comprehension and visual question answering.

Towards Unified Spoken Language Understanding Decoding via Label-aware Compact Linguistics Representations

Aligner²: Enhancing Joint Multiple Intent Detection and Slot Filling Via Adjustive and Forced Cross-Task Alignment

A Joint and Domain-Adaptive Approach to Spoken Language Understanding

Dance with Labels: Dual-Heterogeneous Label Graph Interaction for Multi-intent Spoken Language Understanding.

A Model with Length-Variable Attention for Spoken Language Understanding

SaLa: Scenario-aware Label Graph Interaction for Multi-intent Spoken Language Understanding

Enhancing Reinforcement Learning with Label-Sensitive Reward for Natural Language Understanding

HC$^2$L: Hybrid and Cooperative Contrastive Learning for Cross-lingual Spoken Language Understanding

Learning to Decode Collaboratively with Multiple Language Models

A Self-Attentive Model with Gate Mechanism for Spoken Language Understanding

Joint Spoken Language Understanding And Domain Adaptive Language Modeling

Investigating Decoder-only Large Language Models for Speech-to-text Translation

Incorporating Complete Syntactical Knowledge for Spoken Language Understanding

CoVLM: Composing Visual Entities and Relationships in Large Language Models Via Communicative Decoding

A Hierarchical Decoding Model for Spoken Language Understanding from Unaligned Data.

A Joint Model of Intent Determination and Slot Filling for Spoken Language Understanding.

A Unified Framework for Multi-intent Spoken Language Understanding with prompting

Towards Spoken Language Understanding via Multi-level Multi-grained Contrastive Learning

Joint Modelling of Spoken Language Understanding Tasks with Integrated Dialog History

On decoder-only architecture for speech-to-text and large language model integration

Looking Right is Sometimes Right: Investigating the Capabilities of Decoder-only LLMs for Sequence Labeling