Abstract: Vision-language models are pre-trained by aligning image-text pairs in a common space so that the models can deal with open-set visual concepts by learning semantic information from textual labels. To boost the transferability of these models on downstream tasks in a zero-shot manner, recent works explore generating fixed or learnable prompts, i.e., classification weights are synthesized from natural language describing task-relevant categories, to reduce the gap between tasks in the training and test phases. However, how and what prompts can improve inference performance remains unclear. In this paper, we explicitly provide exploration and clarify the importance of including semantic information in prompts, while existing prompt methods generate prompts without exploring the semantic information of textual labels. A challenging issue is that manually constructing prompts, with rich semantic information, requires domain expertise and is extremely time-consuming. To this end, we propose Causality-pruning Knowledge Prompt (CapKP) for adapting pre-trained vision-language models to downstream image recognition. CapKP retrieves an ontological knowledge graph by treating the textual label as a query to explore task-relevant semantic information. To further refine the derived semantic information, CapKP introduces causality-pruning by following the first principle of Granger causality. Empirically, we conduct extensive evaluations to demonstrate the effectiveness of CapKP, e.g., with 8 shots, CapKP outperforms the manual-prompt method by 12.51% and the learnable-prompt method by 1.39% on average, respectively. Experimental analyses prove the superiority of CapKP in domain generalization compared to benchmark approaches.

TaIncBC: Topic-aware In-context Prompt with Bias Calibration for Event Causality Identification

In-context Contrastive Learning for Event Causality Identification

DAPrompt: Deterministic Assumption Prompt Learning for Event Causality Identification

Causality-aware Concept Extraction based on Knowledge-guided Prompting

Causal Prompting: Debiasing Large Language Model Prompting based on Front-Door Adjustment

Measuring Inductive Biases of In-Context Learning with Underspecified Demonstrations

Bi-Directional Iterative Prompt-Tuning for Event Argument Extraction

Efficient Prompting via Dynamic In-Context Learning

Batch Calibration: Rethinking Calibration for In-Context Learning and Prompt Engineering

TEPrompt: Task Enlightenment Prompt Learning for Implicit Discourse Relation Recognition

PTEKC: pre-training with event knowledge of ConceptNet for cross-lingual event causality identification

Prompt-Based Bias Calibration for Better Zero/Few-Shot Learning of Language Models

Towards Informative Few-Shot Prompt with Maximum Information Gain for In-Context Learning

Eliminating the Language Bias for Visual Question Answering with fine-grained Causal Intervention

Event Causality Identification via Derivative Prompt Joint Learning.

Take Care of Your Prompt Bias! Investigating and Mitigating Prompt Bias in Factual Knowledge Extraction

Topic-aware Causal Intervention for Counterfactual Detection

A Survey of Event Causality Identification: Principles, Taxonomy, Challenges, and Assessment

Supporting Vision-Language Model Inference with Causality-pruning Knowledge Prompt

Learning To Retrieve Prompts for In-Context Learning

More Samples or More Prompts? Exploring Effective In-Context Sampling for LLM Few-Shot Prompt Engineering