Abstract:A remarkable ability of human beings resides in compositional reasoning, i.e., the capacity to make "infinite use of finite means". However, current large vision-language foundation models (VLMs) fall short of such compositional abilities due to their "bag-of-words" behaviors and inability to construct words that correctly represent visual entities and the relations among the entities. To this end, we propose CoVLM, which can guide the LLM to explicitly compose visual entities and relationships among the text and dynamically communicate with the vision encoder and detection network to achieve vision-language communicative decoding. Specifically, we first devise a set of novel communication tokens for the LLM, for dynamic communication between the visual detection system and the language system. A communication token is generated by the LLM following a visual entity or a relation, to inform the detection network to propose regions that are relevant to the sentence generated so far. The proposed regions-of-interests (ROIs) are then fed back into the LLM for better language generation contingent on the relevant regions. The LLM is thus able to compose the visual entities and relationships through the communication tokens. The vision-to-language and language-to-vision communication are iteratively performed until the entire sentence is generated. Our framework seamlessly bridges the gap between visual perception and LLMs and outperforms previous VLMs by a large margin on compositional reasoning benchmarks (e.g., ~20% in HICO-DET mAP, ~14% in Cola top-1 accuracy, and ~3% on ARO top-1 accuracy). We also achieve state-of-the-art performances on traditional vision-language tasks such as referring expression comprehension and visual question answering.

Query-based Composition for Large-Scale Language Model in LVCSR

A Synchronized Pruning Composition Algorithm of Weighted Finite State Transducers for Large Vocabulary Speech Recognition

Linguistic Search Optimization for Deep Learning Based LVCSR

CoVLM: Composing Visual Entities and Relationships in Large Language Models Via Communicative Decoding

Multilingual and Fully Non-Autoregressive ASR with Large Language Model Fusion: A Comprehensive Study

An Asynchronous WFST-Based Decoder for Automatic Speech Recognition

Query Rewriting for Retrieval-Augmented Large Language Models

Query Rewriting in Retrieval-Augmented Large Language Models.

On decoder-only architecture for speech-to-text and large language model integration

Multi-View Lstm Language Model With Word-Synchronized Auxiliary Feature For Lvcsr

Using Large Language Model for End-to-End Chinese ASR and NER

Low Space-Complexity Composition Algorithm for Weighted Finite-State Transducers

Investigating Decoder-only Large Language Models for Speech-to-text Translation

SentenceVAE: Enable Next-sentence Prediction for Large Language Models with Faster Speed, Higher Accuracy and Longer Context

Large-scale Language Model Rescoring on Long-form Data

Connecting Speech Encoder and Large Language Model for ASR

Integrating Multi-Level Linguistic Knowledge with a Unified Framework for Mandarin Speech Recognition

Enhancing Advanced Visual Reasoning Ability of Large Language Models

LLV-FSR: Exploiting Large Language-Vision Prior for Face Super-resolution

ON MODULAR TRAINING OF NEURAL ACOUSTICS-TO-WORD MODEL FOR LVCSR

Video-driven musical composition using large language model with memory-augmented state space