Abstract:Simultaneous sequence generation is a pivotal task for real-time scenarios, such as streaming speech recognition, simultaneous machine translation and simultaneous speech translation, where the target sequence is generated while receiving the source sequence. The crux of achieving high-quality generation with low latency lies in identifying the optimal moments for generating, accomplished by learning a mapping between the source and target sequences. However, existing methods often rely on task-specific heuristics for different sequence types, limiting the model's capacity to adaptively learn the source-target mapping and hindering the exploration of multi-task learning for various simultaneous tasks. In this paper, we propose a unified segment-to-segment framework (Seg2Seg) for simultaneous sequence generation, which learns the mapping in an adaptive and unified manner. During the process of simultaneous generation, the model alternates between waiting for a source segment and generating a target segment, making the segment serve as the natural bridge between the source and target. To accomplish this, Seg2Seg introduces a latent segment as the pivot between source to target and explores all potential source-target mappings via the proposed expectation training, thereby learning the optimal moments for generating. Experiments on multiple simultaneous generation tasks demonstrate that Seg2Seg achieves state-of-the-art performance and exhibits better generality across various tasks.

End-to-end hard constrained text generation via incrementally predicting segments

Controllable Text Generation with Language Constraints

Unlocking Anticipatory Text Generation: A Constrained Approach for Large Language Models Decoding

Intertwining CP and NLP: The Generation of Unreasonably Constrained Sentences

Evaluating, Understanding, and Improving Constrained Text Generation for Large Language Models

A Comprehensive Evaluation of Constrained Text Generation for Large Language Models.

Why is constrained neural language generation particularly challenging?

POINTER: Constrained Progressive Text Generation via Insertion-based Generative Pre-training

Controlled Text Generation with Natural Language Instructions

An Extensible Plug-and-Play Method for Multi-Aspect Controllable Text Generation

Constraints First: A New MDD-based Model to Generate Sentences Under Constraints

Backward and Forward Language Modeling for Constrained Sentence Generation

Toward Controlled Generation of Text

Toward Unified Controllable Text Generation via Regular Expression Instruction

Sentence Semantic Regression for Text Generation

ISF-GAN: Imagine, Select, and Fuse with GPT-Based Text Enrichment for Text-to-Image Synthesis

Unified Segment-to-Segment Framework for Simultaneous Sequence Generation

A Sequence-to-Sequence Approach with Mixed Pointers to Topic Segmentation and Segment Labeling

Simple Techniques for Enhancing Sentence Embeddings in Generative Language Models

Mixture of Soft Prompts for Controllable Data Generation