Abstract:In addition to conveying the linguistic content from source speech to converted speech, maintaining the speaking style of source speech also plays an important role in the voice conversion (VC) task, which is essential in many scenarios with highly expressive source speech, such as dubbing and data augmentation. Previous work generally took explicit prosodic features or fixed-length style embedding extracted from source speech to model the speaking style of source speech, which is insufficient to achieve comprehensive style modeling and target speaker timbre preservation. Inspired by the style's multi-scale nature of human speech, a multi-scale style modeling method for the VC task, referred to as MSM-VC, is proposed in this paper. MSM-VC models the speaking style of source speech from different levels. To effectively convey the speaking style and meanwhile prevent timbre leakage from source speech to converted speech, each level's style is modeled by specific representation. Specifically, prosodic features, pre-trained ASR model's bottleneck features, and features extracted by a model trained with a self-supervised strategy are adopted to model the frame, local, and global-level styles, respectively. Besides, to balance the performance of source style modeling and target speaker timbre preservation, an explicit constraint module consisting of a pre-trained speech emotion recognition model and a speaker classifier is introduced to MSM-VC. This explicit constraint module also makes it possible to simulate the style transfer inference process during the training to improve the disentanglement ability and alleviate the mismatch between training and inference. Experiments performed on the highly expressive speech corpus demonstrate that MSM-VC is superior to the state-of-the-art VC methods for modeling source speech style while maintaining good speech quality and speaker similarity.

Style Modeling for Multi-Speaker Articulation-to-Speech

MSStyleTTS: Multi-Scale Style Modeling with Hierarchical Context Information for Expressive Speech Synthesis

Joint and Adversarial Training with ASR for Expressive Speech Synthesis

Meta-StyleSpeech : Multi-Speaker Adaptive Text-to-Speech Generation

ArtSpeech: Adaptive Text-to-Speech Synthesis with Articulatory Representations

Towards Multi-Scale Style Control for Expressive Speech Synthesis

MSM-VC: High-fidelity Source Style Transfer for Non-Parallel Voice Conversion by Multi-scale Style Modeling

StyleTTS: A Style-Based Generative Model for Natural and Diverse Text-to-Speech Synthesis

Improving the quality of neural TTS using long-form content and multi-speaker multi-style modeling

Towards Multi-Scale Speaking Style Modelling with Hierarchical Context Information for Mandarin Speech Synthesis

Speaking Style Compensation on Synthetic Audio for Robust Keyword Spotting.

Speaker Adaptation on Articulation and Acoustics for Articulation-to-Speech Synthesis

Spoken Style Learning with Multi-modal Hierarchical Context Encoding for Conversational Text-to-Speech Synthesis.

Self-supervised Context-aware Style Representation for Expressive Speech Synthesis

Style Mixture of Experts for Expressive Text-To-Speech Synthesis

Unsupervised Multi-scale Expressive Speaking Style Modeling with Hierarchical Context Information for Audiobook Speech Synthesis.

Multi-speaker Multi-style Text-to-speech Synthesis with Single-speaker Single-style Training Data Scenarios

Style Transfer for Co-Speech Gesture Animation: A Multi-Speaker Conditional-Mixture Approach

Unsupervised Learning for Multi-Style Speech Synthesis with Limited Data

AS-Speech: Adaptive Style for Speech Synthesis