Abstract:The rapid advancement of large language models (LLMs) has significantly propelled the development of text-based chatbots, demonstrating their capability to engage in coherent and contextually relevant dialogues. However, extending these advancements to enable end-to-end speech-to-speech conversation bots remains a formidable challenge, primarily due to the extensive dataset and computational resources required. The conventional approach of cascading automatic speech recognition (ASR), LLM, and text-to-speech (TTS) models in a pipeline, while effective, suffers from unnatural prosody because it lacks direct interactions between the input audio and its transcribed text and the output audio. These systems are also limited by their inherent latency from the ASR process for real-time applications. This paper introduces Style-Talker, an innovative framework that fine-tunes an audio LLM alongside a style-based TTS model for fast spoken dialog generation. Style-Talker takes user input audio and uses transcribed chat history and speech styles to generate both the speaking style and text for the response. Subsequently, the TTS model synthesizes the speech, which is then played back to the user. While the response speech is being played, the input speech undergoes ASR processing to extract the transcription and speaking style, serving as the context for the ensuing dialogue turn. This novel pipeline accelerates the traditional cascade ASR-LLM-TTS systems while integrating rich paralinguistic information from input speech. Our experimental results show that Style-Talker significantly outperforms the conventional cascade and speech-to-speech baselines in terms of both dialogue naturalness and coherence while being more than 50% faster.

PROTOTYPE-TO-STYLE: Dialogue Generation with Style-Aware Editing on Retrieval Memory

Stylistic Retrieval-based Dialogue System with Unparallel Training Data

Stylized Dialogue Response Generation Using Stylized Unpaired Texts

Stylistic Dialogue Generation via Information-Guided Reinforcement Learning Strategy

Stylized Dialogue Generation with Feature-Guided Knowledge Augmentation

Style-Talker: Finetuning Audio Language Model and Style-Based Text-to-Speech Model for Fast Spoken Dialogue Generation

Generating Emotional Social Chatbot Responses with a Consistent Speaking Style

Controlling Style in Generated Dialogue

StyleDGPT: Stylized Response Generation with Pre-trained Language Models

SHADE: Speaker-History-Aware Dialog Generation Through Contrastive and Prompt Learning

Stylized Dialogue Generation with Multi-Pass Dual Learning

Polite Dialogue Generation Without Parallel Data

StyleChat: Learning Recitation-Augmented Memory in LLMs for Stylized Dialogue Generation

StyEmp: Stylizing Empathetic Response Generation via Multi-Grained Prefix Encoder and Personality Reinforcement

A Response Generation Framework Based on Empathy Factors, Common Sense, and Persona

Improving Knowledge-Aware Dialogue Response Generation by Using Human-Written Prototype Dialogues.

Designing Style Matching Conversational Agents

Response Generation by Context-aware Prototype Editing

Structuring Latent Spaces for Stylized Response Generation

Less is More: Learning to Refine Dialogue History for Personalized Dialogue Generation

A Novel Two-Stage Generation Framework for Promoting the Persona-Consistency and Diversity of Responses in Neural Dialog Systems