Abstract:Linking neural representations to linguistic factors is crucial in order to build and analyze NLP models interpretable by humans. Among these factors, syntactic roles (e.g. subjects, direct objects,$\dots$) and their realizations are essential markers since they can be understood as a decomposition of predicative structures and thus the meaning of sentences. Starting from a deep probabilistic generative model with attention, we measure the interaction between latent variables and realizations of syntactic roles and show that it is possible to obtain, without supervision, representations of sentences where different syntactic roles correspond to clearly identified different latent variables. The probabilistic model we propose is an Attention-Driven Variational Autoencoder (ADVAE). Drawing inspiration from Transformer-based machine translation models, ADVAEs enable the analysis of the interactions between latent variables and input tokens through attention. We also develop an evaluation protocol to measure disentanglement with regard to the realizations of syntactic roles. This protocol is based on attention maxima for the encoder and on latent variable perturbations for the decoder. Our experiments on raw English text from the SNLI dataset show that $\textit{i)}$ disentanglement of syntactic roles can be induced without supervision, $\textit{ii)}$ ADVAE separates syntactic roles better than classical sequence VAEs and Transformer VAEs, $\textit{iii)}$ realizations of syntactic roles can be separately modified in sentences by mere intervention on the associated latent variables. Our work constitutes a first step towards unsupervised controllable content generation. The code for our work is publicly available.

Unsupervised Disentanglement Learning Model for Exemplar-Guided Paraphrase Generation

Paraphrase Generation As Unsupervised Machine Translation.

An Adversarial Approach for Unsupervised Syntax-Guided Paraphrase Generation

ParaMac: A General Unsupervised Paraphrase Generation Framework Leveraging Semantic Constraints and Diversifying Mechanisms.

Unsupervised Paraphrasing by Simulated Annealing

Unsupervised Dual Paraphrasing for Two-stage Semantic Parsing

From Paraphrasing to Semantic Parsing: Unsupervised Semantic Parsing Via Synchronous Semantic Decoding

MCPG: A Flexible Multi-Level Controllable Framework for Unsupervised Paraphrase Generation.

Exploring Diverse Expressions for Paraphrase Generation

Multilingual Lexical Simplification via Paraphrase Generation

Towards Unsupervised Content Disentanglement in Sentence Representations via Syntactic Roles

ConRPG: Paraphrase Generation using Contexts as Regularizer

An Empirical Comparison on Imitation Learning and Reinforcement Learning for Paraphrase Generation

Language as a Latent Sequence: deep latent variable models for semi-supervised paraphrase generation

Holistic Exploration on Universal Decompositional Semantic Parsing: Architecture, Data Augmentation, and LLM Paradigm

Integrating Linguistic Knowledge to Sentence Paraphrase Generation.

Query and Output: Generating Words by Querying Distributed Word Representations for Paraphrase Generation

Hierarchical Sketch Induction for Paraphrase Generation

Unsupervised Cross-Lingual Sentence Representation Learning via Linguistic Isomorphism

A Semantically Consistent and Syntactically Variational Encoder-Decoder Framework for Paraphrase Generation.

Paraphrase Generation with Collaboration Between the Forward and the Backward Decoder