Abstract:Recent advancements in spoken dialogue models, exemplified by systems like GPT-4o, have captured significant attention in the speech domain. Compared to traditional three-tier cascaded spoken dialogue models that comprise speech recognition (ASR), large language models (LLMs), and text-to-speech (TTS), modern spoken dialogue models exhibit greater intelligence. These advanced spoken dialogue models not only comprehend audio, music, and other speech-related features, but also capture stylistic and timbral characteristics in speech. Moreover, they generate high-quality, multi-turn speech responses with low latency, enabling real-time interaction through simultaneous listening and speaking capability. Despite the progress in spoken dialogue systems, there is a lack of comprehensive surveys that systematically organize and analyze these systems and the underlying technologies. To address this, we have first compiled existing spoken dialogue systems in the chronological order and categorized them into the cascaded and end-to-end paradigms. We then provide an in-depth overview of the core technologies in spoken dialogue models, covering aspects such as speech representation, training paradigm, streaming, duplex, and interaction capabilities. Each section discusses the limitations of these technologies and outlines considerations for future research. Additionally, we present a thorough review of relevant datasets, evaluation metrics, and benchmarks from the perspectives of training and evaluating spoken dialogue systems. We hope this survey will contribute to advancing both academic research and industrial applications in the field of spoken dialogue systems. The related material is available at <a class="link-external link-https" href="https://github.com/jishengpeng/WavChat" rel="external noopener nofollow">this https URL</a>.

Dialogue Management Systems: a Survey and Overview

A Survey on Dialogue Management in Human-Robot Interaction

Teaching Machines to Converse

Dialogue Management in Conversational Systems: A Review of Approaches, Challenges, and Opportunities

A Review of Dialogue Systems: From Trained Monkeys to Stochastic Parrots

Survey on Evaluation Methods for Dialogue Systems

A Survey of Dialogue System Evaluation.

Survey of evaluation methods for dialogue systems}{Survey of evaluation methods for dialogue systems

Talking with Machines: A Comprehensive Survey of Emergent Dialogue Systems

A Survey on Proactive Dialogue Systems: Problems, Methods, and Prospects

Software-Based Dialogue Systems: Survey, Taxonomy and Challenges

A review of dialogue systems: current trends and future directions

Evaluating Task-oriented Dialogue Systems: A Systematic Review of Measures, Constructs and their Operationalisations

How to Evaluate Your Dialogue Models: A Review of Approaches

A Survey of the Evolution of Language Model-Based Dialogue Systems

Towards a Neural Era in Dialogue Management for Collaboration: A Literature Survey

Don't Forget Your ABC's: Evaluating the State-of-the-Art in Chat-Oriented Dialogue Systems

Let's Negotiate! A Survey of Negotiation Dialogue Systems

A Survey on Multimodal Dialogue Systems: Recent Advances and New Frontiers

A Logic-based Multi-agent System for Ethical Monitoring and Evaluation of Dialogues

WavChat: A Survey of Spoken Dialogue Models