Abstract:Recent work (e.g. LAMA (Petroni et al., 2019)) has found that the quality of the factual information extracted from Large Language Models (LLMs) depends on the prompts used to query them. This inconsistency is problematic because different users will query LLMs for the same information using different wording, but should receive the same, accurate responses regardless. In this work we aim to address this shortcoming by introducing P-Adapters: lightweight models that sit between the embedding layer and first attention layer of LLMs. They take LLM embeddings as input and output continuous prompts that are used to query the LLM. Additionally, we investigate Mixture of Experts (MoE) models that learn a set of continuous prompts ("experts") and select one to query the LLM. They require a separate classifier trained on human-annotated data to map natural language prompts to the continuous ones. P-Adapters perform comparably to the more complex MoE models in extracting factual information from BERT and RoBERTa while eliminating the need for additional annotations. P-Adapters show between 12-26% absolute improvement in precision and 36-50% absolute improvement in consistency over a baseline of only using natural language queries. Finally, we investigate what makes P-Adapters successful and conclude that a significant factor is access to the LLM's embeddings of the original natural language prompt, particularly the subject of the entity pair being queried.

What problem does this paper attempt to address?

The problem that this paper attempts to solve is the sensitivity of large - language models (LLMs) to different query statements when extracting factual information. Specifically, when users use different natural - language query statements to obtain the same information, LLMs may give inconsistent or incorrect answers. This inconsistency is a problem because users expect to be able to obtain the same and accurate answers from LLMs regardless of the wording used. To address this challenge, the paper introduces P - Adapters, which are lightweight models located between the embedding layer and the first attention layer of LLMs. P - Adapters receive the embeddings of LLMs as input and output continuous prompts, which are used to query LLMs. In addition, the paper also explores Mixture of Experts (MoE) models, which learn a set of continuous prompts ("experts") and map natural - language prompts to these continuous prompts through a separately trained classifier. However, MoE models require additional manually - annotated data to train the classifier. The main contribution of P - Adapters lies in their ability to improve the accuracy and consistency of extracting factual information from LLMs without increasing the need for additional annotation. Compared with using only natural - language queries, P - Adapters improve the precision by 12 - 26% and the consistency by 36 - 50%. The paper verifies the effectiveness of P - Adapters through experiments in multiple settings, including In - Domain (ID), Out - of - Distribution Prompts (OOD Prompts), Out - of - Distribution Objects (OOD Objects) and Out - of - Distribution Keyboard Errors (OOD Keyboard Errors). The experimental results show that P - Adapters perform well when dealing with different types of natural - language prompts, especially in maintaining prediction consistency.

P-Adapters: Robustly Extracting Factual Information from Language Models with Diverse Prompts

LLM-Adapters: An Adapter Family for Parameter-Efficient Fine-Tuning of Large Language Models

Generative Adapter: Contextualizing Language Models in Parameters with A Single Forward Pass

Prompt-Aware Adapter: Towards Learning Adaptive Visual Tokens for Multimodal Large Language Models

Adapters for Enhanced Modeling of Multilingual Knowledge and Text

The Effectiveness of Masked Language Modeling and Adapters for Factual Knowledge Injection

The Power of Adaptation: Boosting In-Context Learning through Adaptive Prompting

Experience Adapter: Adapting Pre-trained Language Models for Continual Task Planning.

PERSOMA: PERsonalized SOft ProMpt Adapter Architecture for Personalized Language Prompting

MedAdapter: Efficient Test-Time Adaptation of Large Language Models towards Medical Reasoning

Utilization of pre-trained language models for adapter-based knowledge transfer in software engineering

AdapterSwap: Continuous Training of LLMs with Data Removal and Access-Control Guarantees

PILL: Plug Into LLM with Adapter Expert and Attention Gate

AdvPrompter: Fast Adaptive Adversarial Prompting for LLMs

AdapterEM: Pre-trained Language Model Adaptation for Generalized Entity Matching using Adapter-tuning

BBox-Adapter: Lightweight Adapting for Black-Box Large Language Models

MAPO: Boosting Large Language Model Performance with Model-Adaptive Prompt Optimization

Adapting Multilingual LLMs to Low-Resource Languages with Knowledge Graphs via Adapters

SelfPrompt: Autonomously Evaluating LLM Robustness via Domain-Constrained Knowledge Guidelines and Refined Adversarial Prompts

Long-form factuality in large language models

Making Small Language Models Better Multi-task Learners with Mixture-of-Task-Adapters