Abstract:In natural language generation, most decoding methods are not intrinsic because their performance depends on extrinsically configured hyperparameters. It means that: first, the generation system is dynamic under different conditions while the decoding system is always static under any conditions once its hyperparameters are extrinsically fixed; second, it is hard to select a constant decoding hyperparameter that is omnipotent for all conditions. Although there are decoding methods that are hyperparameter-free, such as greedy and plain sampling, it has been well studied that these methods generally perform worse than methods with hyperparameters, such as beam search, top-k and top-p. Decoding with hyperparameters can get infinite strategies from different fixed configurations, while hyperparameter-free methods have only one strategy. Therefore, the comparison between them is actually unfair, which is a one-vs-infinite battle. So how to deal with the decoding hyperparameters properly and intrinsically? Is it true that hyperparameter-free methods are always inferior to methods with inexhaustible hyperparameter configurations? Is it possible to design a generalized framework, by which these decoding methods can be naturally connected, uniformly described, and mutually inspired? In these paper, we try to find answers to these questions. To this end, we first propose a generalized decoding framework, i.e., GSD, that can be used to uniformly describe and connect existing popular decoding methods. As far as we know, this is the first work trying to build a theoretical framework to associate these decoding methods in formal mathematical theorems. Based on the framework, we then propose Intrinsic Decoding, a novel implementation of GSD with distinctive design from existing decoding algorithms: it is intrinsic and dynamic. Intrinsic Decoding changes the aforementioned comparison from one-vs-infinite to dynamic-vs-infinite. Just like greedy and sampling, Intrinsic Decoding has no hyperparameter, while effecting better than both greedy and sampling, even achieving comparable performance to the methods equipped with inexhaustible hyperparameter configurations, such as beam search, top-k and top-p.

A Simple, Fast Diverse Decoding Algorithm for Neural Generation

Learning to Decode for Future Success

Mutual Information and Diverse Decoding Improve Neural Machine Translation.

Deep Reinforcement Learning for Dialogue Generation

Generating More Interesting Responses in Neural Conversation Models with Distributional Constraints

Generating Diverse Back-Translations Via Constraint Random Decoding

Superposed Decoding: Multiple Generations from a Single Autoregressive Inference Pass

Generating Diverse and High-Quality Texts by Minimum Bayes Risk Decoding

Learning Discourse-level Diversity for Neural Dialog Models using Conditional Variational Autoencoders

IFDID: Information Filter upon Diversity-Improved Decoding for Diversity-Faithfulness Tradeoff in NLG

Follow the Wisdom of the Crowd: Effective Text Generation via Minimum Bayes Risk Decoding

Decoding and Diversity in Machine Translation

Diversifying Neural Conversation Model with Maximal Marginal Relevance.

Improving Diversity of Neural Text Generation Via Inverse Probability Weighting

A Generalized Decoding Method for Neural Text Generation

A Learning-Exploring Method to Generate Diverse Paraphrases with Multi-Objective Deep Reinforcement Learning

Informed Sampling for Diversity in Concept-to-Text NLG

Best-$k$ Search Algorithm for Neural Text Generation

Adaptive Contrastive Search: Uncertainty-Guided Decoding for Open-Ended Text Generation

MEMD: A Diversity-Promoting Learning Framework for Short-Text Conversation.

Bayesian Neural Decoding Using A Diversity-Encouraging Latent Representation Learning Method