Abstract:Image captioning is a longstanding problem in the field of computer vision and natural language processing. To date, researchers have produced impressive state-of-the-art performance in the age of deep learning. Most of these state-of-the-art, however, requires large volume of annotated image-caption pairs in order to train their models. When given an image dataset of interests, practitioner needs to annotate the caption for each image in the training set and this process needs to happen for each newly collected image dataset. In this paper, we explore the task of unsupervised image captioning which utilizes unpaired images and texts to train the model so that the texts can come from different sources than the images. A main school of research on this topic that has been shown to be effective is to construct pairs from the images and texts in the training set according to their overlap of objects. Unlike in the supervised setting, these constructed pairings are however not guaranteed to have fully overlapping set of objects. Our work in this paper overcomes this by harvesting objects corresponding to a given sentence from the training set, even if they don't belong to the same image. When used as input to a transformer, such mixture of objects enables larger if not full object coverage, and when supervised by the corresponding sentence, produced results that outperform current state of the art unsupervised methods by a significant margin. Building upon this finding, we further show that (1) additional information on relationship between objects and attributes of objects also helps in boosting performance; and (2) our method also extends well to non-English image captioning, which usually suffers from a scarcer level of annotations. Our findings are supported by strong empirical results. Our code is available at https://github.com/zihangm/obj-centric-unsup-caption.

Unsupervised Object Discovery Via Object-Centric Representation

Weakly-supervised Contrastive Learning for Unsupervised Object Discovery

Unsupervised Discovery of Object-Centric Neural Fields

Unsupervised Object Discovery and Co-Localization by Deep Descriptor Transforming

Dataset-driven Unsupervised Object Discovery for Region-based Instance Image Retrieval

Towards Unsupervised Object Detection From LiDAR Point Clouds

Universal Object Detection with Large Vision Model

Unsupervised Open-Vocabulary Object Localization in Videos

Unsupervised Part Discovery via Dual Representation Alignment

Object-Centric Unsupervised Image Captioning

Discovery of Topical Object in Image Collections

Masked Multi-Query Slot Attention for Unsupervised Object Discovery

Unsupervised Recognition of Unknown Objects for Open-World Object Detection

Unsupervised Discovery of Object Landmarks as Structural Representations

Zero-Shot Object-Centric Representation Learning

Self-Supervised Object Localization with Joint Graph Partition

Unsupervised Object-Centric Learning from Multiple Unspecified Viewpoints

Self-supervisory Signals for Object Discovery and Detection

Foreground Guidance and Multi-Layer Feature Fusion for Unsupervised Object Discovery with Transformers

Unsupervised 3D category discovery and point labeling from a large urban environment