Abstract:Recent advancements in spoken dialogue models, exemplified by systems like GPT-4o, have captured significant attention in the speech domain. Compared to traditional three-tier cascaded spoken dialogue models that comprise speech recognition (ASR), large language models (LLMs), and text-to-speech (TTS), modern spoken dialogue models exhibit greater intelligence. These advanced spoken dialogue models not only comprehend audio, music, and other speech-related features, but also capture stylistic and timbral characteristics in speech. Moreover, they generate high-quality, multi-turn speech responses with low latency, enabling real-time interaction through simultaneous listening and speaking capability. Despite the progress in spoken dialogue systems, there is a lack of comprehensive surveys that systematically organize and analyze these systems and the underlying technologies. To address this, we have first compiled existing spoken dialogue systems in the chronological order and categorized them into the cascaded and end-to-end paradigms. We then provide an in-depth overview of the core technologies in spoken dialogue models, covering aspects such as speech representation, training paradigm, streaming, duplex, and interaction capabilities. Each section discusses the limitations of these technologies and outlines considerations for future research. Additionally, we present a thorough review of relevant datasets, evaluation metrics, and benchmarks from the perspectives of training and evaluating spoken dialogue systems. We hope this survey will contribute to advancing both academic research and industrial applications in the field of spoken dialogue systems. The related material is available at <a class="link-external link-https" href="https://github.com/jishengpeng/WavChat" rel="external noopener nofollow">this https URL</a>.

Word-class Stochastic Model and Knowledge Representation in a Spoken Language Dialogue System

WORD-CLASS STOCHASTIC MODEL IN A SPOKEN LANGUAGE DIALOGUE SYSTEM

Topic-Oriented Spoken Dialogue Summarization for Customer Service with Saliency-Aware Topic Modeling

A Dialogue System Based on Keyword Spotting Technology

Dynamic Query Organization and Response Generation in Spoken Dialogue System

AN AUTOMATIC SPEECH RECOGNITION STRATEGY DIRECTED BY THE SEMANTIC KNOWLEDGE IN DIALOGUE SYSTEM

Improving the Probabilistic Framework for Representing Dialogue Systems with User Response Model

WavChat: A Survey of Spoken Dialogue Models

KRP-DS: A Knowledge Graph-Based Dialogue System with Inference-Aided Prediction

Towards information-rich, logical dialogue systems with knowledge-enhanced neural models

GLM-Dialog: Noise-tolerant Pre-training for Knowledge-grounded Dialogue Generation

Section-Aware Commonsense Knowledge-Grounded Dialogue Generation with Pre-trained Language Model.

Knowledge Augmented Dialogue Generation with Divergent Facts Selection

Stochastic Language Models for Chinese Speech Recognition Based on Chinese Spelling

A Chinese spoken dialogue system for train information

Spontaneous Speech Parsing in Travel Information Inquiring and Booking Systems

Language Understanding Component for Chinese Dialogue System.

Towards Generalized Models for Task-oriented Dialogue Modeling on Spoken Conversations

Using Textual Interface to Align External Knowledge for End-to-End Task-Oriented Dialogue Systems

Robust Parsing in Spoken Dialogue Systems.

Know Deeper: Knowledge-Conversation Cyclic Utilization Mechanism for Open-domain Dialogue Generation