Abstract:Most existing text generation models follow the sequence-to-sequence paradigm. Generative Grammar suggests that humans generate natural language texts by learning language grammar. We propose a syntax-guided generation schema, which generates the sequence guided by a constituency parse tree in a top-down direction. The decoding process can be decomposed into two parts: (1) predicting the infilling texts for each constituent in the lexicalized syntax context given the source sentence; (2) mapping and expanding each constituent to construct the next-level syntax context. Accordingly, we propose a structural beam search method to find possible syntax structures hierarchically. Experiments on paraphrase generation and machine translation show that the proposed method outperforms autoregressive baselines, while also demonstrating effectiveness in terms of interpretability, controllability, and diversity.
What problem does this paper attempt to address?
### Problems the paper attempts to solve
This paper attempts to solve some key problems in natural language generation (NLG) tasks, especially improving the quality, interpretability, controllability and diversity of the generated text. Specifically, the author proposes a syntax - guided generation schema, aiming to improve the existing autoregressive generation models in the following ways:
1. **Improve generation quality**: By introducing syntactic structures to guide the generation process, reducing generation errors and generating texts that are more in line with grammatical norms.
2. **Enhance interpretability**: Make the generation process more transparent, enabling it to be traced back to specific syntactic component expansion steps, thus making it easier to understand and debug the generation model.
3. **Increase controllability**: Allow flexible control of syntactic components at a specific depth during the generation process, for example, modifying the translation of verb phrases or using syntactic templates to constrain the rewriting style.
4. **Improve diversity**: By exploring multiple syntactic structures in the syntactic tree, generate more diverse text sequences.
### Main contributions of the paper
1. **Propose a syntax - guided generation scheme**: This scheme generates text from top to bottom through the constituency parse tree, which is decomposed into two main parts: predicting the filling text of each syntactic component, and mapping and expanding each syntactic component to build the syntactic context at the next level.
2. **Structured beam search method**: In order to explore more reasonable syntactic structures during the inference process, a structured beam search method is proposed, which can search for the best hypothesis layer by layer in the entire syntactic tree.
3. **Experimental verification**: Through experiments on paraphrase generation and machine translation tasks, the superiority of this method in terms of generation quality, interpretability, controllability and diversity is proved.
### Experimental results
- **Paraphrase generation**: On the ParaNMT - small dataset, compared with the traditional autoregressive Transformer model, this method has achieved significant improvements in multiple metrics, especially in iBLEU, lexical diversity and syntactic diversity.
- **Machine translation**: On multiple benchmark datasets (such as NIST Zh - En, WMT’16 Ro - En, WMT’14 De - En and WMT’14 En - De), this method has obtained higher BLEU scores than the baseline model, especially more obvious in the greedy decoding setting.
### Conclusion
This paper effectively solves multiple key problems in natural language generation tasks by introducing a syntax - guided generation scheme, improving the quality, interpretability, controllability and diversity of the generated text. The experimental results show that this method performs well in both paraphrase generation and machine translation tasks and has broad application prospects.