Abstract:Conversational Speech Synthesis (CSS) aims to express a target utterance with the proper speaking style in a user-agent conversation setting. Existing CSS methods employ effective multi-modal context modeling techniques to achieve empathy understanding and expression. However, they often need to design complex network architectures and meticulously optimize the modules within them. In addition, due to the limitations of small-scale datasets containing scripted recording styles, they often fail to simulate real natural conversational styles. To address the above issues, we propose a novel generative expressive CSS system, termed GPT-Talker.We transform the multimodal information of the multi-turn dialogue history into discrete token sequences and seamlessly integrate them to form a comprehensive user-agent dialogue context. Leveraging the power of GPT, we predict the token sequence, that includes both semantic and style knowledge, of response for the agent. After that, the expressive conversational speech is synthesized by the conversation-enriched VITS to deliver feedback to the user.Furthermore, we propose a large-scale Natural CSS Dataset called NCSSD, that includes both naturally recorded conversational speech in improvised styles and dialogues extracted from TV shows. It encompasses both Chinese and English languages, with a total duration of 236 hours.We conducted comprehensive experiments on the reliability of the NCSSD and the effectiveness of our GPT-Talker. Both subjective and objective evaluations demonstrate that our model outperforms other state-of-the-art CSS systems significantly in terms of naturalness and expressiveness. The Code, Dataset, and Pre-trained Model are available at: <a class="link-external link-https" href="https://github.com/AI-S2-Lab/GPT-Talker" rel="external noopener nofollow">this https URL</a>.

A Context-Based Talker

Teaching Machines to Converse

Uman-in-thel oop

FCTalker: Fine and Coarse Grained Context Modeling for Expressive Conversational Speech Synthesis

Coupled Context Modeling For Deep Chit-Chat: Towards Conversations Between Human And Computer

Context-Controlled Topic-Aware Neural Response Generation for Open-Domain Dialog Systems

A Conversational Model for Eliciting New Chatting Topics in Open-Domain Conversation.

How To Make Context More Useful? An Empirical Study On Context-Aware Neural Conversational Models

Enhancing Speaking Styles in Conversational Text-to-Speech Synthesis with Graph-based Multi-modal Context Modeling

"Shall I Be Your Chat Companion?"

Hard-style Selective Context Utilization for dialogue generation based on what user just said

Contextual understanding by computers

Towards a Better Understanding of Context and Context-Awareness

Speaker or Listener? the Role of a Dialogue Agent

Chat More if You Like: Dynamic Cue Words Planning to Flow Longer Conversations

When Less is More: Using Less Context Information to Generate Better Utterances in Group Conversations.

Introducing the Talk Markup Language (TalkML):Adding a little social intelligence to industrial speech interfaces

CA-BERT: Leveraging Context Awareness for Enhanced Multi-Turn Chat Interaction

Generative Expressive Conversational Speech Synthesis

Talk Less, Interact Better: Evaluating In-context Conversational Adaptation in Multimodal LLMs

Cluster Analysis Based on Contextual Features Extraction for Conversational Corpus