Abstract:In natural language processing, most models try to learn semantic representations merely from texts. The learned representations encode the distributional semantics but fail to connect to any knowledge about the physical world. In contrast, humans learn language by grounding concepts in perception and action and the brain encodes grounded semantics for cognition. Inspired by this notion and recent work in vision-language learning, we design a two-stream model for grounding language learning in vision. The model includes a VGG-based visual stream and a Bert-based language stream. The two streams merge into a joint representational space. Through cross-modal contrastive learning, the model first learns to align visual and language representations with the MS COCO dataset. The model further learns to retrieve visual objects with language queries through a cross-modal attention module and to infer the visual relations between the retrieved objects through a bilinear operator with the Visual Genome dataset. After training, the language stream of this model is a stand-alone language model capable of embedding concepts in a visually grounded semantic space. This semantic space manifests principal dimensions explainable with human intuition and neurobiological knowledge. Word embeddings in this semantic space are predictive of human-defined norms of semantic features and are segregated into perceptually distinctive clusters. Furthermore, the visually grounded language model also enables compositional language understanding based on visual knowledge and multimodal image search with queries based on images, texts, or their combinations.

Learning Zero-Shot Multifaceted Visually Grounded Word Embeddings via Multi-Task Training

Joint Learning of Attended Zero-Shot Features and Visual-Semantic Mapping.

Visual Grounding of Inter-lingual Word-Embeddings

Learning Multi-Modal Word Representation Grounded in Visual Context

Learning to Ground VLMs without Forgetting

Visual Grounding With Joint Multimodal Representation and Interaction

Visual Grounding Strategies for Text-Only Natural Language Processing

Visual Grounding in Video for Unsupervised Word Translation

Embodied multimodal multitask learning

VGSE: Visually-Grounded Semantic Embeddings for Zero-Shot Learning

Seeing the advantage: visually grounding word embeddings to better capture human semantic knowledge

Visual Word2Vec (vis-w2v): Learning Visually Grounded Word Embeddings Using Abstract Scenes

GroundVLP: Harnessing Zero-shot Visual Grounding from Vision-Language Pre-training and Open-Vocabulary Object Detection

Grounding Language Models to Images for Multimodal Inputs and Outputs

Learning semantic sentence representations from visually grounded language without lexical knowledge

Explainable Semantic Space by Grounding Language to Vision with Cross-Modal Contrastive Learning

Expand BERT Representation with Visual Information via Grounded Language Learning with Multimodal Partial Alignment

Beyond Bilingual: Multi-sense Word Embeddings using Multilingual Context

Grounded Language Acquisition From Object and Action Imagery