Abstract:Humans describe complex scenes with compositionality, using simple text descriptions enriched with links and relationships. While vision-language research has aimed to develop models with compositional understanding capabilities, this is not reflected yet in existing datasets which, for the most part, still use plain text to describe images. In this work, we propose a new annotation strategy, graph-based captioning (GBC) that describes an image using a labelled graph structure, with nodes of various types. The nodes in GBC are created using, in a first stage, object detection and dense captioning tools nested recursively to uncover and describe entity nodes, further linked together in a second stage by highlighting, using new types of nodes, compositions and relations among entities. Since all GBC nodes hold plain text descriptions, GBC retains the flexibility found in natural language, but can also encode hierarchical information in its edges. We demonstrate that GBC can be produced automatically, using off-the-shelf multimodal LLMs and open-vocabulary detection models, by building a new dataset, GBC10M, gathering GBC annotations for about 10M images of the CC12M dataset. We use GBC10M to showcase the wealth of node captions uncovered by GBC, as measured with CLIP training. We show that using GBC nodes' annotations -- notably those stored in composition and relation nodes -- results in significant performance boost on downstream models when compared to other dataset formats. To further explore the opportunities provided by GBC, we also propose a new attention mechanism that can leverage the entire GBC graph, with encouraging experimental results that show the extra benefits of incorporating the graph structure. Our datasets are released at \url{<a class="link-external link-https" href="https://huggingface.co/graph-based-captions" rel="external noopener nofollow">this https URL</a>}.

Graph-based bag-of-words for classification

BHoG: Binary Descriptor for Sketch-Based Image Retrieval

Object Recognition Based on the Region of Interest and Optimal Bag of Words Model.

From visual words to a visual grammar: using language modelling for image classification

A Graph Community and Bag of Categorized Visual Words Based Image Retrieval.

Visual Words Assignment on A Graph Via Minimal Mutual Information Loss

Image classification by visual bag-of-words refinement and reduction

BoW-MSN: A Novel Texture Classification Approach.

Towards Optimal Bag-of-features for Object Categorization and Semantic Video Retrieval

Evaluating Inverted Files for Visual Compact Codes on a Large Scale

Classification-Then-Grounding: Reformulating Video Scene Graphs As Temporal Bipartite Graphs

Understanding bag-of-words model: a statistical framework

Object Recognition Using Words Model of Optimal Size in Histograms of Oriented Gradients

Bag-of-visual-words and Spatial Extensions for Land-Use Classification

A New Method for Graph-Based Representation of Text in Natural Language Processing

Open-World Visual Recognition Using Knowledge Graphs

Exploring Local Features and the Bag-of-Visual-Words Approach for Bioimage Classification.

Graph-Based Captioning: Enhancing Visual Descriptions by Interconnecting Region Captions

Modified Bag of Visual Words Model for Image Classification

Sketch-Based Image Retrieval with a Novel BoVW Representation.

On Embedding Implementations in Text Ranking and Classification Employing Graphs