Abstract:The text-based speech editor allows the editing of speech through intuitive cutting, copying, and pasting operations to speed up the process of editing speech. However, the major drawback of current systems is that edited speech often sounds unnatural due to cut-copy-paste operation. In addition, it is not obvious how to synthesize records according to a new word not appearing in the transcript. This paper proposes a novel end-to-end text-based speech editing method called context-aware mask prediction network (CampNet). The model can simulate the text-based speech editing process by randomly masking part of speech and then predicting the masked region by sensing the speech context. It can solve unnatural prosody in the edited region and synthesize the speech corresponding to the unseen words in the transcript. Secondly, for the possible operation of text-based speech editing, we design three text-based operations based on CampNet: deletion, insertion, and replacement. These operations can cover various situations of speech editing. Thirdly, to synthesize the speech corresponding to long text in insertion and replacement operations, a word-level autoregressive generation method is proposed. Fourthly, we propose a speaker adaptation method using only one sentence for CampNet and explore the ability of few-shot learning based on CampNet, which provides a new idea for speech forgery tasks. The subjective and objective experiments on VCTK and LibriTTS datasets show that the speech editing results based on CampNet are better than TTS technology, manual editing, and VoCo method. We also conduct detailed ablation experiments to explore the effect of the CampNet structure on its performance. Finally, the experiment shows that speaker adaptation with only one sentence can further improve the naturalness of speech. Examples of generated speech can be found at <a class="link-external link-https" href="https://hairuo55.github.io/CampNet" rel="external noopener nofollow">this https URL</a>.

FluentEditor: Text-based Speech Editing by Considering Acoustic and Prosody Consistency

FluentEditor2: Text-based Speech Editing by Modeling Multi-Scale Acoustic and Prosody Consistency

FluentEditor+: Text-based Speech Editing by Modeling Local Hierarchical Acoustic Smoothness and Global Prosody Consistency

FluentSpeech: Stutter-Oriented Automatic Speech Editing with Context-Aware Diffusion Models

DiffEditor: Enhancing Speech Editing with Semantic Enrichment and Acoustic Consistency

AudioEditor: A Training-Free Diffusion-Based Audio Editing Framework

Speech Editing -- a Summary

E $^{3}$ TTS: End-to-End Text-Based Speech Editing TTS System and Its Applications

Context-Aware Prosody Correction for Text-Based Speech Editing

Listen, Chat, and Edit: Text-Guided Soundscape Modification for Enhanced Auditory Experience

End-to-end Code-switched TTS with Mix of Monolingual Recordings.

Fine-Grained and Interpretable Neural Speech Editing

AttentionStitch: How Attention Solves the Speech Editing Problem

Context-Aware Mask Prediction Network for End-to-End Text-Based Speech Editing

Adaptive Text to Speech for Spontaneous Style

CampNet: Context-Aware Mask Prediction for End-to-End Text-Based Speech Editing

FCTalker: Fine and Coarse Grained Context Modeling for Expressive Conversational Speech Synthesis

AdapEdit: Spatio-Temporal Guided Adaptive Editing Algorithm for Text-Based Continuity-Sensitive Image Editing

Fluent and Low-latency Simultaneous Speech-to-Speech Translation with Self-adaptive Training

Text-based editing of talking-head video

Enhancing Emotional Text-to-Speech Controllability with Natural Language Guidance through Contrastive Learning and Diffusion Models