Abstract:Incorporating external knowledge into the response generation process is essential to building more helpful and reliable dialog agents. However, collecting knowledge-grounded conversations is often costly, calling for a better pre-trained model for grounded dialog generation that generalizes well w.r.t. different types of knowledge. In this work, we propose KPT (Keyword-guided Pre-Training), a novel self-supervised pre-training method for grounded dialog generation without relying on extra knowledge annotation. Specifically, we use a pre-trained language model to extract the most uncertain tokens in the dialog as keywords. With these keywords, we construct two kinds of knowledge and pre-train a knowledge-grounded response generation model, aiming at handling two different scenarios: (1) the knowledge should be faithfully grounded; (2) it can be selectively used. For the former, the grounding knowledge consists of keywords extracted from the response. For the latter, the grounding knowledge is additionally augmented with keywords extracted from other utterances in the same dialog. Since the knowledge is extracted from the dialog itself, KPT can be easily performed on a large volume and variety of dialogue data. We considered three data sources (open-domain, task-oriented, conversational QA) with a total of 2.5M dialogues. We conduct extensive experiments on various few-shot knowledge-grounded generation tasks, including grounding on dialog acts, knowledge graphs, persona descriptions, and Wikipedia passages. Our comprehensive experiments and analyses demonstrate that KPT consistently outperforms state-of-the-art methods on these tasks with diverse grounding knowledge.

Are Pre-trained Language Models Knowledgeable to Ground Open Domain Dialogues?

Knowledge-Grounded Dialogue Generation with Pre-trained Language Models

Knowledge Grounded Pre-Trained Model for Dialogue Response Generation.

Improving Dialogue Generation via Proactively Querying Grounded Knowledge

GLM-Dialog: Noise-tolerant Pre-training for Knowledge-grounded Dialogue Generation

A Knowledge Driven Dialogue Model with Reinforcement Learning

An Empirical Investigation of Pre-Trained Transformer Language Models for Open-Domain Dialogue Generation

Eliciting Knowledge from Large Pre-Trained Models for Unsupervised Knowledge-Grounded Conversation

Knowledge-Grounded Dialogue with Reward-Driven Knowledge Selection

Improving Dialog System Grounded with Unstructured Knowledge by Domain Adaptive Pre-Training and Post-Ranking

Section-Aware Commonsense Knowledge-Grounded Dialogue Generation with Pre-trained Language Model.

KRP-DS: A Knowledge Graph-Based Dialogue System with Inference-Aided Prediction

KPT: Keyword-guided Pre-training for Grounded Dialog Generation

Can Generative Pre-trained Language Models Serve As Knowledge Bases for Closed-book QA?

Infusing internalized knowledge of language models into hybrid prompts for knowledgeable dialogue generation

Zero-Resource Knowledge-Grounded Dialogue Generation

Pan More Gold from the Sand: Refining Open-domain Dialogue Training with Noisy Self-Retrieval Generation

A Pre-training Strategy for Zero-Resource Response Selection in Knowledge-Grounded Conversations

Prediction, Selection, and Generation: Exploration of Knowledge-Driven Conversation System

Do Pre-trained Models Benefit Knowledge Graph Completion? A Reliable Evaluation and a Reasonable Approach.

Low-Resource Knowledge-Grounded Dialogue Generation