Abstract:Most previous work on trainable language generation has focused on two paradigms: (a) using a generation decisions of an existing generator. Both approaches rely on the existence of a handcrafted generation component, which is likely to limit their scalability to new domains. The first contribution of this article is to present Bagel, a fully data-driven generation method that treats the language generation task as a search for the most likely sequence of semantic concepts and realization phrases, according to Factored Language Models (FLMs). As domain utterances are not readily available for most natural language generation tasks, a large creative effort is required to produce the data necessary to represent human linguistic variation for nontrivial domains. This article is based on the assumption that learning to produce paraphrases can be facilitated by collecting data from a large sample of untrained annotators using crowdsourcing—rather than a few domain experts—by relying on a coarse meaning representation. A second contribution of this article is to use crowdsourced data to show how dialogue naturalness can be improved by learning to vary the output utterances generated for a given semantic input. Two data-driven methods for generating paraphrases in dialogue are presented: (a) by sampling from the n-best list of realizations produced by Bagel's FLM reranker; and (b) by learning a structured perceptron predicting whether candidate realizations are valid paraphrases. We train Bagel on a set of 1,956 utterances produced by 137 annotators, which covers 10 types of dialogue acts and 128 semantic concepts in a tourist information system for Cambridge. An automated evaluation shows that Bagel outperforms utterance class LM baselines on this domain. A human evaluation of 600 resynthesized dialogue extracts shows that Bagel's FLM output produces utterances comparable to a handcrafted baseline, whereas the perceptron classifier performs worse. Interestingly, human judges find the system sampling from the n-best list to be more natural than a system always returning the first-best utterance. The judges are also more willing to interact with the n-best system in the future. These results suggest that capturing the large variation found in human language using data-driven methods is beneficial for dialogue interaction.

Probing BERT's priors with serial reproduction chains

Distilling Knowledge Learned in BERT for Text Generation

Evaluating Models of Robust Word Recognition with Serial Reproduction

BERTs are Generative In-Context Learners

Mole-BERT: Rethinking Pre-training Graph Neural Networks for Molecules

Promises and Pitfalls of Generative Masked Language Modeling: Theoretical Framework and Practical Guidelines

Transparency at the Source: Evaluating and Interpreting Language Models With Access to the True Distribution

Eliciting the Priors of Large Language Models using Iterated In-Context Learning

Inconsistencies in Masked Language Models

Closing the Curious Case of Neural Text Degeneration

Probabilistically Masked Language Model Capable of Autoregressive Generation in Arbitrary Word Order

Comparing Abstraction in Humans and Large Language Models Using Multimodal Serial Reproduction

BERT has a Mouth, and It Must Speak: BERT as a Markov Random Field Language Model

Solving a Cloze Test for Generative Commonsense Question Answering.

Investigating Masking-based Data Generation in Language Models

Probing Pretrained Language Models for Lexical Semantics

Weighted Sampling for Masked Language Modeling

Tokens, the oft-overlooked appetizer: Large language models, the distributional hypothesis, and meaning

Stochastic Language Generation in Dialogue using Factored Language Models

Do Syntactic Probes Probe Syntax? Experiments with Jabberwocky Probing

On the Inductive Bias of Masked Language Modeling: From Statistical to Syntactic Dependencies