Abstract:Recently, two approaches, fine-tuning large pre-trained language models and variational training, have attracted significant interests, separately, for semi-supervised end-to-end task-oriented dialog (TOD) systems. In this paper, we propose Variational Latent-State GPT model (VLS-GPT), which is the first to combine the strengths of the two approaches. Among many options of models, we propose the generative model and the inference model for variational learning of the end-to-end TOD system, both as auto-regressive language models based on GPT-2, which can be further trained over a mix of labeled and unlabeled dialog data in a semi-supervised manner. Variational training of VLS-GPT is both statistically and computationally more challenging than previous variational learning works for sequential latent variable models, which use turn-level first-order Markovian. The inference model in VLS-GPT is non-Markovian due to the use of the Transformer architecture. In this work, we establish Recursive Monte Carlo Approximation (RMCA) to the variational objective with non-Markovian inference model and prove its unbiasedness. Further, we develop the computational strategy of sampling-then-forward-computation to realize RMCA, which successfully overcomes the memory explosion issue of using GPT in variational learning and speeds up training. Semi-supervised TOD experiments are conducted on two benchmark multi-domain datasets of different languages - MultiWOZ2.1 and CrossWOZ. VLS-GPT is shown to significantly outperform both supervised-only and semi-supervised self-training baselines.

A Latent Variable Model with Hierarchical Structure and GPT-2 for Long Text Generation

A Transformer-Based Hierarchical Variational AutoEncoder Combined Hidden Markov Model for Long Text Generation

A Hierarchical Neural Autoencoder for Paragraphs and Documents

AdaVAE: Exploring Adaptive GPT-2s in Variational Auto-Encoders for Language Modeling

Long and Diverse Text Generation with Planning-based Hierarchical Variational Model

Improve Variational Autoencoder for Text Generationwith Discrete Latent Bottleneck

Text Generation Based on Generative Adversarial Nets with Latent Variable

Text Adversarial Generation Based on Latent Variable Models

APo-VAE: Text Generation in Hyperbolic Space

Generative Text Modeling Through Short Run Inference

Dispersed EM-VAEs for Interpretable Text Generation

A Temporal Variational Model for Story Generation

Topic-Guided Variational Auto-Encoder for Text Generation.

Fuse It More Deeply! A Variational Transformer with Layer-Wise Latent Variable Inference for Text Generation

HierTTS: Expressive End-to-End Text-to-Waveform Using a Multi-Scale Hierarchical Variational Auto-Encoder

Latent-Glat: Glancing at Latent Variables for Parallel Text Generation

A Hybrid Convolutional Variational Autoencoder for Text Generation

Dispersed Exponential Family Mixture VAEs for Interpretable Text Generation

Optimus: Organizing Sentences via Pre-trained Modeling of a Latent Space

Hierarchical Generative Modeling for Controllable Speech Synthesis

Variational Latent-State GPT for Semi-Supervised Task-Oriented Dialog Systems