Abstract:Streaming automatic speech recognition (ASR) aims to emit each hypothesized word as quickly and accurately as possible. However, emitting fast without degrading quality, as measured by word error rate (WER), is highly challenging. Existing approaches including Early and Late Penalties [1] and Constrained Alignments [2], [3] penalize emission delay by manipulating per-token or per-frame probability prediction in sequence transducer models [4]. While being successful in reducing delay, these approaches suffer from significant accuracy regression and also require additional word alignment information from an existing model. In this work, we propose a sequence-level emission regularization method, named FastEmit, that applies latency regularization directly on per-sequence probability in training transducer models, and does not require any alignment. We demonstrate that FastEmit is more suitable to the sequence-level optimization of transducer models [4] for streaming ASR by applying it on various end-to-end streaming ASR networks including RNN-Transducer [5], Transformer-Transducer [6], [7], ConvNet-Transducer [8] and Conformer-Transducer [9]. We achieve 150 ~ 300ms latency reduction with significantly better accuracy over previous techniques on a Voice Search test set. FastEmit also improves streaming ASR accuracy from 4.4%/8.9% to 3.1%/7.5% WER, meanwhile reduces 90th percentile latency from 210ms to only 30ms on LibriSpeech.

Low Latency ASR for Simultaneous Speech Translation

End-to-End Evaluation for Low-Latency Simultaneous Speech Translation

Divergence-Guided Simultaneous Speech Translation

From Start to Finish: Latency Reduction Strategies for Incremental Speech Synthesis in Simultaneous Speech-to-Speech Translation

STACL: Simultaneous Translation with Integrated Anticipation and Controllable Latency

Learning When to Speak: Latency and Quality Trade-offs for Simultaneous Speech-to-Speech Translation with Offline Models

Fluent and Low-latency Simultaneous Speech-to-Speech Translation with Self-adaptive Training

Learning Adaptive Segmentation Policy for Simultaneous Translation

Textless Streaming Speech-to-Speech Translation using Semantic Speech Tokens

Label-Synchronous Neural Transducer for E2E Simultaneous Speech Translation

On Efficient Coupling of ASR and SMT for Speech Translation

What does it take to get state of the art in simultaneous speech-to-speech translation?

Re-Translation Strategies for Long Form, Simultaneous, Spoken Language Translation

R-BI: Regularized Batched Inputs enhance Incremental Decoding Framework for Low-Latency Simultaneous Speech Translation

SimulMT to SimulST: Adapting Simultaneous Text Translation to End-to-End Simultaneous Speech Translation

FastEmit: Low-Latency Streaming ASR with Sequence-Level Emission Regularization

Towards Achieving Human Parity on End-to-end Simultaneous Speech Translation via LLM Agent

CA*: Addressing Evaluation Pitfalls in Computation-Aware Latency for Simultaneous Speech Translation

Diffusion Synthesizer for Efficient Multilingual Speech to Speech Translation

Building Accurate Low Latency ASR for Streaming Voice Search

A Study on Low-Latency Recognition-Synthesis-Based Any-to-One Voice Conversion