Abstract:Direct speech-to-speech translation (S2ST) translates speech from one language into another using a single model. However, due to the presence of linguistic and acoustic diversity, the target speech follows a complex multimodal distribution, posing challenges to achieving both high-quality translations and fast decoding speeds for S2ST models. In this paper, we propose DASpeech, a non-autoregressive direct S2ST model which realizes both fast and high-quality S2ST. To better capture the complex distribution of the target speech, DASpeech adopts the two-pass architecture to decompose the generation process into two steps, where a linguistic decoder first generates the target text, and an acoustic decoder then generates the target speech based on the hidden states of the linguistic decoder. Specifically, we use the decoder of DA-Transformer as the linguistic decoder, and use FastSpeech 2 as the acoustic decoder. DA-Transformer models translations with a directed acyclic graph (DAG). To consider all potential paths in the DAG during training, we calculate the expected hidden states for each target token via dynamic programming, and feed them into the acoustic decoder to predict the target mel-spectrogram. During inference, we select the most probable path and take hidden states on that path as input to the acoustic decoder. Experiments on the CVSS Fr-En benchmark demonstrate that DASpeech can achieve comparable or even better performance than the state-of-the-art S2ST model Translatotron 2, while preserving up to 18.53x speedup compared to the autoregressive baseline. Compared with the previous non-autoregressive S2ST model, DASpeech does not rely on knowledge distillation and iterative decoding, achieving significant improvements in both translation quality and decoding speed. Furthermore, DASpeech shows the ability to preserve the speaker's voice of the source speech during translation.

CTC-based Non-autoregressive Textless Speech-to-Speech Translation

CTC-based Non-autoregressive Speech Translation

Non-Autoregressive Transformer ASR with CTC-Enhanced Decoder Input

A Non-autoregressive Generation Framework for End-to-End Simultaneous Speech-to-Speech Translation

Synchronous Speech Recognition and Speech-to-Text Translation with Interactive Decoding.

A Non-autoregressive Generation Framework for End-to-End Simultaneous Speech-to-Any Translation.

DASpeech: Directed Acyclic Transformer for Fast and High-quality Speech-to-Speech Translation

LV-CTC: Non-autoregressive ASR with CTC and latent variable models

Improving Non-Autoregressive End-to-End Speech Recognition with Pre-Trained Acoustic and Language Models

Can We Achieve High-quality Direct Speech-to-Speech Translation without Parallel Speech Data?

Consecutive Decoding for Speech-to-text Translation

CTC-GMM: CTC guided modality matching for fast and accurate streaming speech translation

Investigating the Reordering Capability in CTC-based Non-Autoregressive End-to-End Speech Translation.

AV-TranSpeech: Audio-Visual Robust Speech-to-Speech Translation

Textless Speech-to-Speech Translation With Limited Parallel Data

Back Translation for Speech-to-text Translation Without Transcripts

Cross Attention Augmented Transducer Networks for Simultaneous Translation.

Hybrid Autoregressive and Non-Autoregressive Transformer Models for Speech Recognition

Improving Non-autoregressive Translation Quality with Pretrained Language Model, Embedding Distillation and Upsampling Strategy for CTC

Improving Non-Autoregressive Sign Language Translation with Random Ordering Progressive Prediction Pretraining

Pre-training for Speech Translation: CTC Meets Optimal Transport