Abstract:Recently, zero-shot image captioning has gained increasing attention, where only text data is available for training. The remarkable progress in text-to-image diffusion model presents the potential to resolve this task by employing synthetic image-caption pairs generated by this pre-trained prior. Nonetheless, the defective details in the salient regions of the synthetic images introduce semantic misalignment between the synthetic image and text, leading to compromised results. To address this challenge, we propose a novel Patch-wise Cross-modal feature Mix-up (PCM) mechanism to adaptively mitigate the unfaithful contents in a fine-grained manner during training, which can be integrated into most of encoder-decoder frameworks, introducing our PCM-Net. Specifically, for each input image, salient visual concepts in the image are first detected considering the image-text similarity in CLIP space. Next, the patch-wise visual features of the input image are selectively fused with the textual features of the salient visual concepts, leading to a mixed-up feature map with less defective content. Finally, a visual-semantic encoder is exploited to refine the derived feature map, which is further incorporated into the sentence decoder for caption generation. Additionally, to facilitate the model training with synthetic data, a novel CLIP-weighted cross-entropy loss is devised to prioritize the high-quality image-text pairs over the low-quality counterparts. Extensive experiments on MSCOCO and Flickr30k datasets demonstrate the superiority of our PCM-Net compared with state-of-the-art VLMs-based approaches. It is noteworthy that our PCM-Net ranks first in both in-domain and cross-domain zero-shot image captioning. The synthetic dataset SynthImgCap and code are available at https://jianjieluo.github.io/SynthImgCap.

Zero-shot Image Captioning by Anchor-augmented Vision-Language Space Alignment

Mining Fine-Grained Image-Text Alignment for Zero-Shot Captioning via Text-Only Training

From Association to Generation: Text-only Captioning by Unsupervised Cross-modal Mapping

DeCap: Decoding CLIP Latents for Zero-Shot Captioning via Text-Only Training

Zero-shot Referring Expression Comprehension via Structural Similarity Between Images and Captions

Zero-TextCap: Zero-shot Framework for Text-based Image Captioning

MultiCapCLIP: Auto-Encoding Prompts for Zero-Shot Multilingual Visual Captioning

MeaCap: Memory-Augmented Zero-shot Image Captioning

Language-Driven Cross-Modal Classifier for Zero-Shot Multi-Label Image Recognition

Zoom-shot: Fast and Efficient Unsupervised Zero-Shot Transfer of CLIP to Vision Encoders with Multimodal Loss

Unleashing Text-to-Image Diffusion Prior for Zero-Shot Image Captioning

Zero-Shot Audio Captioning Using Soft and Hard Prompts

Entrocap: Zero-Shot Image Captioning with Entropy-Based Retrieval

Zero-shot Visual Question Answering with Language Model Feedback

Modeling Caption Diversity in Contrastive Vision-Language Pretraining

CLIPS: An Enhanced CLIP Framework for Learning with Synthetic Captions

Exploring Diverse In-Context Configurations for Image Captioning

CLIP Models are Few-shot Learners: Empirical Studies on VQA and Visual Entailment

An Eye for an Ear: Zero-shot Audio Description Leveraging an Image Captioner using Audiovisual Distribution Alignment

Language-Driven Visual Consensus for Zero-Shot Semantic Segmentation