Abstract:Incorporating external knowledge into the response generation process is essential to building more helpful and reliable dialog agents. However, collecting knowledge-grounded conversations is often costly, calling for a better pre-trained model for grounded dialog generation that generalizes well w.r.t. different types of knowledge. In this work, we propose KPT (Keyword-guided Pre-Training), a novel self-supervised pre-training method for grounded dialog generation without relying on extra knowledge annotation. Specifically, we use a pre-trained language model to extract the most uncertain tokens in the dialog as keywords. With these keywords, we construct two kinds of knowledge and pre-train a knowledge-grounded response generation model, aiming at handling two different scenarios: (1) the knowledge should be faithfully grounded; (2) it can be selectively used. For the former, the grounding knowledge consists of keywords extracted from the response. For the latter, the grounding knowledge is additionally augmented with keywords extracted from other utterances in the same dialog. Since the knowledge is extracted from the dialog itself, KPT can be easily performed on a large volume and variety of dialogue data. We considered three data sources (open-domain, task-oriented, conversational QA) with a total of 2.5M dialogues. We conduct extensive experiments on various few-shot knowledge-grounded generation tasks, including grounding on dialog acts, knowledge graphs, persona descriptions, and Wikipedia passages. Our comprehensive experiments and analyses demonstrate that KPT consistently outperforms state-of-the-art methods on these tasks with diverse grounding knowledge.

What problem does this paper attempt to address?

The problem that this paper attempts to solve is how to effectively utilize external knowledge in dialogue generation to build more useful and reliable dialogue agents. Specifically, the paper focuses on how to improve the performance of dialogue systems on different types of external knowledge through self - supervised pre - training methods in the absence of additional knowledge annotations. This involves two main challenges: 1. **Diverse knowledge types**: Downstream tasks may require different types of knowledge, such as common - sense knowledge graphs, character descriptions, Wikipedia articles, etc. 2. **Lack of large - scale annotated data**: Collecting dialogue data with knowledge annotations is usually costly, which limits the training and generalization ability of the model. To solve these problems, the paper proposes KPT (Keyword - guided Pre - Training), a new self - supervised pre - training method. It extracts keywords from dialogues as pseudo - knowledge, thus not requiring additional knowledge annotations. KPT aims to handle two different scenarios: 1. **Faithful to given knowledge**: In this case, the model needs to generate responses based on the given knowledge. KPT uses the keywords extracted from the responses as knowledge. 2. **Selective use of knowledge**: In this case, the model can selectively use some or all of the knowledge according to the context. KPT uses randomly sampled keywords from other turns of the dialogue, and (or not including) the keywords extracted from the responses as knowledge. In this way, KPT can be pre - trained on large - scale dialogue data and perform well in multiple downstream tasks, especially in the case of a small number of samples. Experimental results show that KPT significantly outperforms the existing state - of - the - art methods in dialogue generation tasks with multiple knowledge sources.

KPT: Keyword-guided Pre-training for Grounded Dialog Generation

Improving Dialog System Grounded with Unstructured Knowledge by Domain Adaptive Pre-Training and Post-Ranking

PK-ICR: Persona-Knowledge Interactive Multi-Context Retrieval for Grounded Dialogue

Structure Pre-training and Prompt Tuning for Knowledge Graph Transfer

Read Key Points: Dialogue-Grounded Knowledge Points Generation with Multi-Level Salience-Aware Mixture

Knowledge-Grounded Dialogue Generation with Pre-trained Language Models

KGGPT: Empowering Robots with OpenAI's ChatGPT and Knowledge Graph.

Multi-domain Knowledge Graph Collaborative Pre-training and Prompt Tuning for Diverse Downstream Tasks

Augmenting Knowledge-grounded Conversations with Sequential Knowledge Transition.

An empirical study of pre-trained language models in simple knowledge graph question answering

KRP-DS: A Knowledge Graph-Based Dialogue System with Inference-Aided Prediction

Infusing internalized knowledge of language models into hybrid prompts for knowledgeable dialogue generation

Generative Knowledge Selection for Knowledge-Grounded Dialogues

Grounding Dialogue Systems via Knowledge Graph Aware Decoding with Pre-trained Transformers

Are Pre-trained Language Models Knowledgeable to Ground Open Domain Dialogues?

Pretrain-KGE - Learning Knowledge Representation from Pretrained Language Models.

Select, Extend, and Generate: Generative Knowledge Selection for Open-Domain Dialogue Response Generation.

Section-Aware Commonsense Knowledge-Grounded Dialogue Generation with Pre-trained Language Model.

Bridging the Gap Between Prior and Posterior Knowledge Selection for Knowledge-Grounded Dialogue Generation

Do Pre-trained Models Benefit Knowledge Graph Completion? A Reliable Evaluation and a Reasonable Approach.