Improving Zero-shot Voice Style Transfer via Disentangled Representation Learning

Siyang Yuan,Pengyu Cheng,Ruiyi Zhang,Weituo Hao,Zhe Gan,Lawrence Carin

DOI: https://doi.org/10.48550/arXiv.2103.09420

2021-03-17

Abstract:Voice style transfer, also called voice conversion, seeks to modify one speaker's voice to generate speech as if it came from another (target) speaker. Previous works have made progress on voice conversion with parallel training data and pre-known speakers. However, zero-shot voice style transfer, which learns from non-parallel data and generates voices for previously unseen speakers, remains a challenging problem. We propose a novel zero-shot voice transfer method via disentangled representation learning. The proposed method first encodes speaker-related style and voice content of each input voice into separated low-dimensional embedding spaces, and then transfers to a new voice by combining the source content embedding and target style embedding through a decoder. With information-theoretic guidance, the style and content embedding spaces are representative and (ideally) independent of each other. On real-world VCTK datasets, our method outperforms other baselines and obtains state-of-the-art results in terms of transfer accuracy and voice naturalness for voice style transfer experiments under both many-to-many and zero-shot setups.

Audio and Speech Processing,Sound

What problem does this paper attempt to address?

The problem that this paper attempts to solve is zero - shot voice style transfer. Specifically, it is to generate the speech of unseen speakers without parallel training data. Traditional voice conversion methods usually require parallel data sets, that is, paired audio of two speakers speaking the same sentence, which is often difficult to obtain in practical applications. In addition, most of the existing non - parallel data processing methods cannot achieve many - to - many conversions, that is, they cannot convert the voices of multiple source speakers into the styles of multiple target speakers.

The paper proposes a new zero - shot voice conversion method to achieve this goal through disentangled representation learning. This method first encodes the speaker - related style and voice content in each input voice into separate low - dimensional embedding spaces, and then generates new voices by combining the source content embedding and the target style embedding through a decoder. Guided by information theory, the style and content embedding spaces are representative and, ideally, independent of each other. Experimental results show that on the real - world VCTK data set, this method outperforms other baseline methods in terms of both conversion accuracy and speech naturalness in the voice style transfer tasks in many - to - many and zero - shot settings, reaching the state - of - the - art level.