A Picture is Worth a Thousand Words: A Unified System for Diverse Captions and Rich Images Generation

Yupan Huang,Bei Liu,Jianlong Fu,Yutong Lu

DOI: https://doi.org/10.1145/3474085.3478561

2021-10-19

Abstract:A creative image-and-text generative AI system mimics humans' extraordinary abilities to provide users with diverse and comprehensive caption suggestions, as well as rich image creations. In this work, we demonstrate such an AI creation system to produce both diverse captions and rich images. When users imagine an image and associate it with multiple captions, our system paints a rich image to reflect all captions faithfully. Likewise, when users upload an image, our system depicts it with multiple diverse captions. We propose a unified multi-modal framework to achieve this goal. Specifically, our framework jointly models image-and-text representations with a Transformer network, which supports rich image creation by accepting multiple captions as input. We consider the relations among input captions to encourage diversity in training and adopt a non-autoregressive decoding strategy to enable real-time inference. Based on these, our system supports both diverse captions and rich images generations. Our code is available online.

Computer Vision and Pattern Recognition,Computation and Language,Multimedia

What problem does this paper attempt to address?

The problems that this paper attempts to solve are two main issues in the bidirectional image - text generation tasks:

1. **Ambiguous Correspondence between a Single Image and a Single Description**: Traditional image - to - text generation tasks usually assume that an image can only have one accurate description, which ignores the richness and diversity of the image. In fact, a picture may contain multiple perspectives or details, which can be captured by different descriptions. Therefore, it is often insufficient to use a single description to interpret an image.

2. **Insufficient Fine - Grained Alignment between Image and Text**: In image - to - text or text - to - image generation, a single image - text pair cannot provide accurate and fine - grained alignment between objects and semantic units. This alignment is crucial for the learning of image and text representations.

To solve the above problems, the author proposes a bidirectional image - text generation framework that can generate multiple diverse descriptions as well as rich images. Specifically, this framework is based on a unified multimodal Transformer network. It supports rich image generation by accepting multiple descriptions as input and considers the relationships between input descriptions during the training process to encourage diversity. In addition, the framework adopts a non - autoregressive decoding strategy to achieve real - time inference. In this way, the system can not only generate multiple descriptions from a single image but also generate an image that reflects all descriptions from multiple descriptions. The experimental results on the MSCOCO Captions dataset show that this method outperforms the baseline models in generating diverse descriptions and high - quality images.