Abstract:Over the past year, the field of Natural Language Generation (NLG) has experienced an exponential surge, largely due to the introduction of Large Language Models (LLMs). These models have exhibited the most effective performance in a range of domains within the Natural Language Processing and Generation domains. However, their application in domain-specific tasks, such as paraphrasing, presents significant challenges. The extensive number of parameters makes them difficult to operate on commercial hardware, and they require substantial time for inference, leading to high costs in a production setting. In this study, we tackle these obstacles by employing LLMs to develop three distinct models for the paraphrasing field, applying a method referred to as sequence-level knowledge distillation. These distilled models are capable of maintaining the quality of paraphrases generated by the LLM. They demonstrate faster inference times and the ability to generate diverse paraphrases of comparable quality. A notable characteristic of these models is their ability to exhibit syntactic diversity while also preserving lexical diversity, features previously uncommon due to existing data quality issues in datasets and not typically observed in neural-based approaches. Human evaluation of our models shows that there is only a 4% drop in performance compared to the LLM teacher model used in the distillation process, despite being 1000 times smaller. This research provides a significant contribution to the NLG field, offering a more efficient and cost-effective solution for paraphrasing tasks.

Document-level paraphrase generation base on attention enhanced graph LSTM

Towards Document-Level Paraphrase Generation with Sentence Rewriting and Reordering.

A Hierarchical Neural Autoencoder for Paragraphs and Documents

ParaMac: A General Unsupervised Paraphrase Generation Framework Leveraging Semantic Constraints and Diversifying Mechanisms.

A Deep Generative Framework for Paraphrase Generation

Towards Diverse Paraphrase Generation Using Multi-Class Wasserstein GAN

Diversified Paraphrase Generation with Commonsense Knowledge Graph

Query and Output: Generating Words by Querying Distributed Word Representations for Paraphrase Generation

WORD EMBEDDING ATTENTION NETWORK: GENERATING WORDS BY QUERYING DISTRIBUTED WORD REPRESENTATIONS FOR PARAPHRASE GENERATION

Pushing Paraphrase Away from Original Sentence: A Multi-Round Paraphrase Generation Approach.

Bidirectional Long Short-Term Memory with Gated Relevance Network for Paraphrase Identification

DivGAN: Towards Diverse Paraphrase Generation via Diversified Generative Adversarial Network.

Semantic-guided attention and adaptive gating for document-level relation extraction

Paraphrase Generation Model Integrating Transformer Architecture, Part-of-Speech Features, and Pointer Generator Network

Not All Metrics Are Guilty: Improving NLG Evaluation with LLM Paraphrasing

Generating Diverse and Descriptive Image Captions Using Visual Paraphrases

VTechAGP: An Academic-to-General-Audience Text Paraphrase Dataset and Benchmark Models

Joint Copying and Restricted Generation for Paraphrase.

Parameter Efficient Diverse Paraphrase Generation Using Sequence-Level Knowledge Distillation

Exploring Effective Inter-Encoder Semantic Interaction for Document-Level Relation Extraction

Abstractive Document Summarization with a Graph-Based Attentional Neural Model.