Abstract:Recent advancements in spoken dialogue models, exemplified by systems like GPT-4o, have captured significant attention in the speech domain. Compared to traditional three-tier cascaded spoken dialogue models that comprise speech recognition (ASR), large language models (LLMs), and text-to-speech (TTS), modern spoken dialogue models exhibit greater intelligence. These advanced spoken dialogue models not only comprehend audio, music, and other speech-related features, but also capture stylistic and timbral characteristics in speech. Moreover, they generate high-quality, multi-turn speech responses with low latency, enabling real-time interaction through simultaneous listening and speaking capability. Despite the progress in spoken dialogue systems, there is a lack of comprehensive surveys that systematically organize and analyze these systems and the underlying technologies. To address this, we have first compiled existing spoken dialogue systems in the chronological order and categorized them into the cascaded and end-to-end paradigms. We then provide an in-depth overview of the core technologies in spoken dialogue models, covering aspects such as speech representation, training paradigm, streaming, duplex, and interaction capabilities. Each section discusses the limitations of these technologies and outlines considerations for future research. Additionally, we present a thorough review of relevant datasets, evaluation metrics, and benchmarks from the perspectives of training and evaluating spoken dialogue systems. We hope this survey will contribute to advancing both academic research and industrial applications in the field of spoken dialogue systems. The related material is available at <a class="link-external link-https" href="https://github.com/jishengpeng/WavChat" rel="external noopener nofollow">this https URL</a>.

The Eighth Dialog System Technology Challenge

Overview of the Eighth Dialog System Technology Challenge: DSTC8

Overview of the Ninth Dialog System Technology Challenge: DSTC9

Overview of the Tenth Dialog System Technology Challenge: DSTC10

Multi-domain Task-oriented Dialog Challenge II

Results of the Multi-Domain Task-Completion Dialog Challenge

Microsoft Dialogue Challenge: Building End-to-End Task-Completion Dialogue Systems

Deep Contextualized Utterance Representations for Response Selection and Dialogue Analysis

The SPPD System for Schema Guided Dialogue State Tracking Challenge

Taskmaster-1: Toward a Realistic and Diverse Dialog Dataset

WavChat: A Survey of Spoken Dialogue Models

Recent Advances and Challenges in Task-Oriented Dialog Systems

Can Current Task-oriented Dialogue Models Automate Real-world Scenarios in the Wild?

The Dialogue Dodecathlon: Open-Domain Knowledge and Image Grounded Conversational Agents

Intent Induction from Conversations for Task-Oriented Dialogue Track at DSTC 11

Comparing Data Augmentation Methods for End-to-End Task-Oriented Dialog Systems

Dialogos: a robust system for human-machine spoken dialogue on the telephone

Learning to Retrieve Entity-Aware Knowledge and Generate Responses with Copy Mechanism for Task-Oriented Dialogue Systems

Challenges in Building Intelligent Open-domain Dialog Systems

CHAT: a Conversational Helper for Automotive Tasks

Adapting Text-based Dialogue State Tracker for Spoken Dialogues