Abstract:Word embeddings, i.e., semantically meaningful vector representation of words, are largely influenced by the distributional hypothesis "You shall know a word by the company it keeps" (Harris, 1954), whereas modern prediction-based neural network embeddings rely on design choices and hyperparameter optimization. Word embeddings like Word2Vec, GloVe etc. well capture the contextuality and real-world analogies but contemporary convolution-based image embeddings such as VGGNet, AlexNet, etc. do not capture contextual knowledge. The popular king-queen analogy does not hold true for most commonly used vision embeddings. In this paper, we introduce a pre-trained joint embedding (JE), named IMAGINATOR, trained on 21K distinct image objects level from 1M image+text pairs. JE is a way to encode multimodal data into a vector space where the text modality serves as the ground-ing key, which the complementary modality (in this case, the image) is anchored with. IMAGINATOR encapsulates three individual representations: (i) object-object co-location, (ii) word-object co-location, and (iii) word-object correlation. These three ways capture complementary aspects of the two modalities which are further combined to obtain the final JEs. Generated JEs are intrinsically evaluated to assess how well they capture the contextuality and real-world analogies. We also evaluate pre-trained IMAGINATOR JEs on three downstream tasks: (i) image captioning, (ii) Image2Tweet, and (iii) text-based image retrieval. IMAGINATOR establishes a new standard on the aforementioned down-stream tasks by outperforming the current SoTA on all the selected tasks. IMAGINATOR will be made publicly available. The codes are available at <a class="link-external link-https" href="https://github.com/varunakk/IMAGINATOR" rel="external noopener nofollow">this https URL</a>

ImageInWords: Unlocking Hyper-Detailed Image Descriptions

Image Textualization: An Automatic Framework for Creating Accurate and Detailed Image Descriptions

An Image is Worth One Word: Personalizing Text-to-Image Generation using Textual Inversion

From Captions to Visual Concepts and Back

DOCCI: Descriptions of Connected and Contrasting Images

Picturing Ambiguity: A Visual Twist on the Winograd Schema Challenge

IIITD-20K: Dense captioning for Text-Image ReID

IMAGINATOR: Pre-Trained Image+Text Joint Embeddings using Word-Level Grounding of Images

Descriptive Image Quality Assessment in the Wild

Enhancing Image Description Generation through Deep Reinforcement Learning: Fusing Multiple Visual Features and Reward Mechanisms

Description Enhancement of Generated Images via Automatic Visual Question Generation and Answering

Getting it Right: Improving Spatial Consistency in Text-to-Image Models

Visual Word2Vec (vis-w2v): Learning Visually Grounded Word Embeddings Using Abstract Scenes

AI Illustrator: Translating Raw Descriptions into Images by Prompt-based Cross-Modal Generation

Generating Images from Spoken Descriptions.

A Picture May Be Worth a Hundred Words for Visual Question Answering

A Picture is Worth More Than 77 Text Tokens: Evaluating CLIP-Style Models on Dense Captions

Hypernymy Understanding Evaluation of Text-to-Image Models via WordNet Hierarchy

Visual Spatial Description: Controlled Spatial-Oriented Image-to-Text Generation

Synthesizing Spoken Descriptions of Images

A Picture is Worth a Thousand Words: A Unified System for Diverse Captions and Rich Images Generation