TokSing: Singing Voice Synthesis based on Discrete Tokens

Yuning Wu,Chunlei zhang,Jiatong Shi,Yuxun Tang,Shan Yang,Qin Jin

2024-06-21

Abstract:Recent advancements in speech synthesis witness significant benefits by leveraging discrete tokens extracted from self-supervised learning (SSL) models. Discrete tokens offer higher storage efficiency and greater operability in intermediate representations compared to traditional continuous Mel spectrograms. However, when it comes to singing voice synthesis(SVS), achieving higher levels of melody expression poses a great challenge for utilizing discrete tokens. In this paper, we introduce TokSing, a discrete-based SVS system equipped with a token formulator that offers flexible token blendings. We observe a melody degradation during discretization, prompting us to integrate a melody signal with the discrete token and incorporate a specially-designed melody enhancement strategy in the musical encoder. Extensive experiments demonstrate that our TokSing achieves better performance against the Mel spectrogram baselines while offering advantages in intermediate representation space cost and convergence speed.

Sound,Audio and Speech Processing

What problem does this paper attempt to address?

The main problem that this paper attempts to solve is how to achieve high - quality melody expression while maintaining efficient storage and flexible operation in a discrete - token - based Singing Voice Synthesis (SVS) system. Specifically, the paper mainly addresses the following two challenges:

1. **Finding a token representation method suitable for singing voice synthesis**:
- Due to copyright restrictions and strict recording environments, there are currently no self - supervised learning (SSL) models specifically for singing. Existing SSL models contain less singing data, so setting appropriate tokens for singing synthesis is a challenge.
- Singing tasks not only need to handle lyrics, which are naturally discrete information, but also need to accurately express melody changes, such as the subtle changes of the same note in different frames, especially in the case of advanced vocal techniques such as long notes, high notes, and trills.

2. **Constructing a discrete - token - based SVS system that can meet the requirements of melody expression**:
- During the discretization process, detailed information of the melody may be lost, resulting in a decline in the quality of the generated singing voice. To this end, the paper proposes a method of fusing melody signals and designs a special melody enhancement strategy to compensate for the melody details that may be lost during the discretization process.

To address these challenges, the paper proposes a new framework named TokSing. This framework achieves better singing voice synthesis effects by introducing flexible token combination strategies and melody control signals. Experimental results show that TokSing is not only superior to the traditional continuous Mel - spectrogram baseline system in performance, but also has advantages in the space cost of intermediate representations and convergence speed.

### Formula summary
- **Melody loss function** \( L_m \):
\[
L_m=\frac{1}{N}\sum_{i = 1}^{N}|m_i-\hat{m}_i|
\]
where \( N \) is the number of frames, \( m_i \) is the logarithm of the fundamental frequency extracted from the original audio, and \(\hat{m}_i\) is the predicted logarithm of the fundamental frequency.

- **Token prediction loss function** \( L_{\text{tok}} \):
\[
L_{\text{tok}}=-\frac{1}{N}\sum_{i = 1}^{N}\sum_{j = 1}^{M}d_{ij}\log(\hat{d}_{ij})
\]
where \( N \) is the number of frames, \( M \) is the number of token layers, \( d_{ij} \) is the real token, and \(\hat{d}_{ij}\) is the predicted token.

Through these improvements, TokSing can significantly improve the quality of singing voice synthesis while ensuring efficient storage and flexible operation.