Abstract:Image captioning bridges the gap between vision and language by automatically generating natural language descriptions for images. Traditional image captioning methods often overlook the preferences and characteristics of users. Personalized image captioning solves this problem by incorporating user prior knowledge into the model, such as writing styles and preferred vocabularies. Most existing methods emphasize the user context fusion process by memory networks or transformers. However, these methods ignore the distinct domains of each dataset. Therefore, they need to update the entire caption model parameters when meeting new samples, which is time-consuming and calculation-intensive. To address this challenge, we propose a novel personalized image captioning framework that leverages user context to consider personality factors. Additionally, our framework utilizes the prefix-tuning paradigm to extract knowledge from a frozen large language model, reducing the gap between different language domains. Specifically, we employ CLIP to extract the visual features of an image and align the semantic space using a query-guided mapping network. By incorporating the transformer layer, we merge the visual features with the user's contextual prior knowledge to generate informative prefixes. Moreover, we employ GPT-2 as the frozen large language model. With a small number of parameters to be trained, our model performs efficiently and effectively. Our model outperforms existing baseline models on Instagram and YFCC100M datasets across five evaluation metrics, demonstrating its superiority, including twofold improvements in metrics such as BLEU-4 and CIDEr.

Controllable Natural Language Generation with Contrastive Prefixes

Control Prefixes for Parameter-Efficient Text Generation

PrefixMol: Target- and Chemistry-aware Molecule Design Via Prefix Embedding

User-Aware Prefix-Tuning is a Good Learner for Personalized Image Captioning

An Extensible Plug-and-Play Method for Multi-Aspect Controllable Text Generation

Focused Prefix Tuning for Controllable Text Generation

Controlling the Focus of Pretrained Language Generation Models

CCPrefix: Counterfactual Contrastive Prefix-Tuning for Many-Class Classification

Large Language Models can Contrastively Refine their Generation for Better Sentence Representation Learning

Prefix-Tuning Based Unsupervised Text Style Transfer

Controllable Text Generation with Language Constraints

Controlled Text Generation for Large Language Model with Dynamic Attribute Graphs

Prefix Propagation: Parameter-Efficient Tuning for Long Sequences

Control Large Language Models via Divide and Conquer

Learning Easily Updated General Purpose Text Representations with Adaptable Task-Specific Prefixes

Few-Shot Table-to-Text Generation with Prefix-Controlled Generator

Prompt-Based Length Controlled Generation with Multiple Control Types

KnowPrefix-Tuning: A Two-Stage Prefix-Tuning Framework for Knowledge-Grounded Dialogue Generation

Towards Infinite-Long Prefix in Transformer

Unlocking Anticipatory Text Generation: A Constrained Approach for Large Language Models Decoding

Large Language Model as Attributed Training Data Generator: A Tale of Diversity and Bias