Abstract:Recent advances on text-to-image generation have witnessed the rise of diffusion models which act as powerful generative models. Nevertheless, it is not trivial to exploit such latent variable models to capture the dependency among discrete words and meanwhile pursue complex visual-language alignment in image captioning. In this paper, we break the deeply rooted conventions in learning Transformer-based encoder-decoder, and propose a new diffusion model based paradigm tailored for image captioning, namely Semantic-Conditional Diffusion Networks (SCD-Net). Technically, for each input image, we first search the semantically relevant sentences via cross-modal retrieval model to convey the comprehensive semantic information. The rich semantics are further regarded as semantic prior to trigger the learning of Diffusion Transformer, which produces the output sentence in a diffusion process. In SCD-Net, multiple Diffusion Transformer structures are stacked to progressively strengthen the output sentence with better visional-language alignment and linguistical coherence in a cascaded manner. Furthermore, to stabilize the diffusion process, a new self-critical sequence training strategy is designed to guide the learning of SCD-Net with the knowledge of a standard autoregressive Transformer model. Extensive experiments on COCO dataset demonstrate the promising potential of using diffusion models in the challenging image captioning task. Source code is available at \url{<a class="link-external link-https" href="https://github.com/YehLi/xmodaler/tree/master/configs/image_caption/scdnet" rel="external noopener nofollow">this https URL</a>}.

Fast Semantic Diffusion for Large-Scale Context-Based Image and Video Annotation

Domain Adaptive Semantic Diffusion for Large Scale Context-Based Video Annotation

A Semantic Context Model For Automatic Image Annotation

Semantic Context Transfer Across Heterogeneous Sources for Domain Adaptive Video Search.

Exploiting Semantic And Visual Context For Effective Video Annotation

Context-based visual concept detection using domain adaptive semantic diffusion

Semantic Context Based Refinement for News Video Annotation.

Open-Vocabulary 3D Semantic Segmentation with Text-to-Image Diffusion Models

Semantic Annotation for Complex Video Street Views Based on 2D–3D Multi-Feature Fusion and Aggregated Boosting Decision Forests

Semantic context learning with large-scale weakly-labeled image set.

Fast Contextual Scene Graph Generation with Unbiased Context Augmentation.

Fast Semantic Image Segmentation with High Order Context and Guided Filtering.

Unsupervised Semantic Correspondence Using Stable Diffusion

Efficient Region-Aware Large Graph Construction Towards Scalable Multi-Label Propagation

Diffusion Features to Bridge Domain Gap for Semantic Segmentation

Semantic-Guided Generative Image Augmentation Method with Diffusion Models for Image Classification

Semantic-Conditional Diffusion Networks for Image Captioning

Context Propagation from Proposals for Semantic Video Object Segmentation

Context Diffusion: In-Context Aware Image Generation

SPDiffusion: Semantic Protection Diffusion for Multi-concept Text-to-image Generation

Distributed Image Understanding with Semantic Dictionary and Semantic Expansion