Abstract:Diffusion-based generative models have exhibited remarkable capability in the production of high-fidelity visual content such as images and videos. However, their performance is significantly contingent upon the quality of textual inputs, commonly referred to as ‘'prompts'. The process of traditional prompt engineering necessitates empirical exper-tise and poses challenges for inexperienced users. In this paper, we introduce PromptCoT, an innovative enhancer that autonomously refines prompts for users. PromptCoT is designed based on the observation that prompts, which re-semble the textual information of high-quality images during training, lead to superior generation performance. Therefore, we fine-tune the Large Language Models (LLM) using a curated text dataset that comprises descriptions of high-quality visual content. Consequently, the LLM can capture the distribution of high-quality texts, enabling it to boost the original texts. Nonetheless, one drawback of LLMs is their tendency to generate irrelevant information. We employ a tailored Chain-of-Thought (CoT) mechanism to address the problem. Our CoT can extract and amalgamate crucial information from the prompt candidates, enabling a reasonable process based on the contextual cues to produce a more comprehensive and nuanced output. Considering computational efficiency, instead of allocating a dedicated LLM to each individual model or dataset, we integrate adapters that facil-itate task-specific adaptation, leveraging a shared LLM as the foundation for this process. With independent fine-tuning of adapters, we can adapt PromptCoT to new datasets while minimally increasing training costs and memory usage. We evaluate the effectiveness of PromptCoT by assessing on widely-used latent diffusion models for visual generation. The results demonstrate significant improvements in key performance metrics.

A Composable Generative Framework Based on Prompt Learning for Various Information Extraction Tasks

A Unified Generative Framework based on Prompt Learning for Various Information Extraction Tasks

OpenPrompt: an Open-source Framework for Prompt-learning

Promoting Unified Generative Framework with Descriptive Prompts for Joint Multi-Intent Detection and Slot Filling

Instance-aware Prompt Learning for Language Understanding and Generation

LightNER: A Lightweight Generative Framework with Prompt-guided Attention for Low-resource NER

On Conditional and Compositional Language Model Differentiable Prompting

TransPrompt - Towards an Automatic Transferable Prompting Framework for Few-shot Text Classification.

Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processing

Contextualized Hybrid Prompt-Tuning for Generation-Based Event Extraction.

HGPROMPT: Bridging Homogeneous and Heterogeneous Graphs for Few-shot Prompt Learning

Prompt-Based Modality Bridging for Unified Text-to-Face Generation and Manipulation

TransPrompt V2: Transferable Prompt-based Fine-tuning for Few-shot Text Classification

A User-Friendly Framework for Generating Model-Preferred Prompts in Text-to-Image Synthesis

TransPrompt v2: A Transferable Prompting Framework for Cross-task Text Classification

Set-Based Prompting: Provably Solving the Language Model Order Dependency Problem

Temporal Prompt Engineering for Generative Semantic Communication

KC-Prompt: End-To-End Knowledge-Complementary Prompting for Rehearsal-Free Continual Learning

KnowPrompt: Knowledge-aware Prompt-tuning with Synergistic Optimization for Relation Extraction

PromptCoT: Align Prompt Distribution Via Adapted Chain-of-Thought

PromptNER: Prompt Locating and Typing for Named Entity Recognition