Introduction to the Issue on Advances in Spoken Dialogue Systems and Mobile Interface

Jason D. Williams,Kai Yu,Brahim Chaib-draa,Oliver Lemon,Roberto Pieraccini,Olivier Pietquin,Pascal Poupart,Steve Young

DOI: https://doi.org/10.1109/jstsp.2012.2234401

IF: 7.695

2012-01-01

IEEE Journal of Selected Topics in Signal Processing

Abstract:The eight papers in this special issue cover advances in spoken dialogue systems and mobile interface.

What problem does this paper attempt to address?

Editorial: Recent advances in intelligent robotic systems

Xin Xu,Yong Liu

DOI: https://doi.org/10.1049/trit.2017.0019

IF: 7.985

2017-01-01

CAAI Transactions on Intelligence Technology

Abstract:This Editorial introduces the motivation and organization process of the special issue on intelligent robotic systems. The major contributions of the six papers published in the special issue are discussed. In these papers, three papers present recent progresses in robot navigation and localization in complex environments. One paper reports a new robot sensing approach. The other two papers address the problems in robot dynamics modeling and service modeling in human-robot interaction.
Special issue on multimodal processing and robotics for dialogue systems (Part II)

David Traum,Gabriel Skantze,Hiromitsu Nishizaki,Ryuichiro Higashinaka,Takashi Minato,Takayuki Nagai

DOI: https://doi.org/10.1080/01691864.2024.2319945

IF: 2.057

2024-03-20

Advanced Robotics

Abstract:The proliferation of devices like Google Home and Amazon Alexa is ongoing, yet the realization of human-like interactions remains unachieved. In particular, dialogue robots and multimodal dialogue systems, which must consider multimodal information and the surrounding environment, require further technological advancements. Towards such advancements, dialogue robot competitions and dialogue system live competitions have been conducted in recent years. Unlike previous dialogue systems that rely solely on verbal language, systems that express through facial expressions and gestures, grounded in the real world, hold the potential to significantly transform society in the future. This special issue is devoted to the progress of such dialogue robots and multimodal dialogue systems.

robotics
Teaching Machines to Converse

Li Jiwei

DOI: https://doi.org/10.48550/arxiv.2001.11701

2020-01-01

Abstract: The ability of a machine to communicate with humans has long been associated with the general success of AI. This dates back to Alan Turing's epoch-making work in the early 1950s, which proposes that a machine's intelligence can be tested by how well it, the machine, can fool a human into believing that the machine is a human through dialogue conversations. Many systems learn generation rules from a minimal set of authored rules or labels on top of hand-coded rules or templates, and thus are both expensive and difficult to extend to open-domain scenarios. Recently, the emergence of neural network models the potential to solve many of the problems in dialogue learning that earlier systems cannot tackle: the end-to-end neural frameworks offer the promise of scalability and language-independence, together with the ability to track the dialogue state and then mapping between states and dialogue actions in a way not possible with conventional systems. On the other hand, neural systems bring about new challenges: they tend to output dull and generic responses; they lack a consistent or a coherent persona; they are usually optimized through single-turn conversations and are incapable of handling the long-term success of a conversation; and they are not able to take the advantage of the interactions with humans. This dissertation attempts to tackle these challenges: Contributions are two-fold: (1) we address new challenges presented by neural network models in open-domain dialogue generation systems; (2) we develop interactive question-answering dialogue systems by (a) giving the agent the ability to ask questions and (b) training a conversation agent through interactions with humans in an online fashion, where a bot improves through communicating with humans and learning from the mistakes that it makes.
A Summary of Research on Intelligent Dialogue Systems

Bangjing Wang,Hui Cao

DOI: https://doi.org/10.1088/1742-6596/1651/1/012020

2020-11-01

Journal of Physics: Conference Series

Abstract:Abstract Dialogue systems have always been an important research direction of natural language processing. In recent years, the potential and commercial value of dialogue systems have attracted people’s attention, and technologies such as intelligent customer service and chatbots have gradually become popular. First, it outlines the concept and classification of the dialog system; then introduces the constituent modules of the dialog system, and mainly studies the key technologies and mainstream ideas; finally, summarizes the problems encountered at this stage and looks forward to the future research direction.
Dialogue with Robots: Proposals for Broadening Participation and Research in the SLIVAR Community

Casey Kennington,Malihe Alikhani,Heather Pon-Barry,Katherine Atwell,Yonatan Bisk,Daniel Fried,Felix Gervits,Zhao Han,Mert Inan,Michael Johnston,Raj Korpan,Diane Litman,Matthew Marge,Cynthia Matuszek,Ross Mead,Shiwali Mohan,Raymond Mooney,Natalie Parde,Jivko Sinapov,Angela Stewart,Matthew Stone,Stefanie Tellex,Tom Williams

2024-04-01

Abstract:The ability to interact with machines using natural human language is becoming not just commonplace, but expected. The next step is not just text interfaces, but speech interfaces and not just with computers, but with all machines including robots. In this paper, we chronicle the recent history of this growing field of spoken dialogue with robots and offer the community three proposals, the first focused on education, the second on benchmarks, and the third on the modeling of language when it comes to spoken interaction with robots. The three proposals should act as white papers for any researcher to take and build upon.

Computation and Language,Robotics
WavChat: A Survey of Spoken Dialogue Models

Shengpeng Ji,Yifu Chen,Minghui Fang,Jialong Zuo,Jingyu Lu,Hanting Wang,Ziyue Jiang,Long Zhou,Shujie Liu,Xize Cheng,Xiaoda Yang,Zehan Wang,Qian Yang,Jian Li,Yidi Jiang,Jingzhen He,Yunfei Chu,Jin Xu,Zhou Zhao

2024-11-15

Abstract:Recent advancements in spoken dialogue models, exemplified by systems like GPT-4o, have captured significant attention in the speech domain. Compared to traditional three-tier cascaded spoken dialogue models that comprise speech recognition (ASR), large language models (LLMs), and text-to-speech (TTS), modern spoken dialogue models exhibit greater intelligence. These advanced spoken dialogue models not only comprehend audio, music, and other speech-related features, but also capture stylistic and timbral characteristics in speech. Moreover, they generate high-quality, multi-turn speech responses with low latency, enabling real-time interaction through simultaneous listening and speaking capability. Despite the progress in spoken dialogue systems, there is a lack of comprehensive surveys that systematically organize and analyze these systems and the underlying technologies. To address this, we have first compiled existing spoken dialogue systems in the chronological order and categorized them into the cascaded and end-to-end paradigms. We then provide an in-depth overview of the core technologies in spoken dialogue models, covering aspects such as speech representation, training paradigm, streaming, duplex, and interaction capabilities. Each section discusses the limitations of these technologies and outlines considerations for future research. Additionally, we present a thorough review of relevant datasets, evaluation metrics, and benchmarks from the perspectives of training and evaluating spoken dialogue systems. We hope this survey will contribute to advancing both academic research and industrial applications in the field of spoken dialogue systems. The related material is available at <a class="link-external link-https" href="https://github.com/jishengpeng/WavChat" rel="external noopener nofollow">this https URL</a>.

Audio and Speech Processing,Computation and Language,Machine Learning,Multimedia,Sound
Talking with Machines: A Comprehensive Survey of Emergent Dialogue Systems

William Tholke

DOI: https://doi.org/10.48550/arXiv.2305.16324

2023-05-10

Abstract:From the earliest experiments in the 20th century to the utilization of large language models and transformers, dialogue systems research has continued to evolve, playing crucial roles in numerous fields. This paper offers a comprehensive review of these systems, tracing their historical development and examining their fundamental operations. We analyze popular and emerging datasets for training and survey key contributions in dialogue systems research, including traditional systems and advanced machine learning methods. Finally, we consider conventional and transformer-based evaluation metrics, followed by a short discussion of prevailing challenges and future prospects in the field.

Computation and Language
Recent Advances and Challenges in Task-Oriented Dialog Systems

Zheng Zhang,Ryuichi Takanobu,Qi Zhu,MinLie Huang,XiaoYan Zhu

DOI: https://doi.org/10.1007/s11431-020-1692-3

2020-01-01

Abstract:Due to the significance and value in human-computer interaction and naturallanguage processing, task-oriented dialog systems are attracting more and moreattention in both academic and industrial communities. In this paper, we surveyrecent advances and challenges in task-oriented dialog systems. We also discussthree critical topics for task-oriented dialog systems: (1) improving dataefficiency to facilitate dialog modeling in low-resource settings, (2) modelingmulti-turn dynamics for dialog policy learning to achieve bettertask-completion performance, and (3) integrating domain ontology knowledge intothe dialog model. Besides, we review the recent progresses in dialog evaluationand some widely-used corpora. We believe that this survey, though incomplete,can shed a light on future research in task-oriented dialog systems.
Challenges in Building Intelligent Open-domain Dialog Systems

Minlie Huang,Xiaoyan Zhu,Jianfeng Gao

DOI: https://doi.org/10.48550/arXiv.1905.05709

2020-02-28

Abstract:There is a resurgent interest in developing intelligent open-domain dialog systems due to the availability of large amounts of conversational data and the recent progress on neural approaches to conversational AI. Unlike traditional task-oriented bots, an open-domain dialog system aims to establish long-term connections with users by satisfying the human need for communication, affection, and social belonging. This paper reviews the recent works on neural approaches that are devoted to addressing three challenges in developing such systems: semantics, consistency, and interactiveness. Semantics requires a dialog system to not only understand the content of the dialog but also identify user's social needs during the conversation. Consistency requires the system to demonstrate a consistent personality to win users trust and gain their long-term confidence. Interactiveness refers to the system's ability to generate interpersonal responses to achieve particular social goals such as entertainment, conforming, and task completion. The works we select to present here is based on our unique views and are by no means complete. Nevertheless, we hope that the discussion will inspire new research in developing more intelligent dialog systems.

Computation and Language,Artificial Intelligence
The State of Speech in HCI: Trends, Themes and Challenges

Leigh Clark,Philip Doyle,Diego Garaialde,Emer Gilmartin,Stephan Schlögl,Jens Edlund,Matthew Aylett,João Cabral,Cosmin Munteanu,Justin Edwards,Benjamin R Cowan

DOI: https://doi.org/10.1093/iwc/iwz016

2019-06-01

Interacting with Computers

Abstract:Abstract Speech interfaces are growing in popularity. Through a review of 99 research papers this work maps the trends, themes, findings and methods of empirical research on speech interfaces in the field of human–computer interaction (HCI). We find that studies are usability/theory-focused or explore wider system experiences, evaluating Wizard of Oz, prototypes or developed systems. Measuring task and interaction was common, as was using self-report questionnaires to measure concepts like usability and user attitudes. A thematic analysis of the research found that speech HCI work focuses on nine key topics: system speech production, design insight, modality comparison, experiences with interactive voice response systems, assistive technology and accessibility, user speech production, using speech technology for development, peoples’ experiences with intelligent personal assistants and how user memory affects speech interface interaction. From these insights we identify gaps and challenges in speech research, notably taking into account technological advancements, the need to develop theories of speech interface interaction, grow critical mass in this domain, increase design work and expand research from single to multiple user interaction contexts so as to reflect current use contexts. We also highlight the need to improve measure reliability, validity and consistency, in the wild deployment and reduce barriers to building fully functional speech interfaces for research. RESEARCH HIGHLIGHTS Most papers focused on usability/theory-based or wider system experience research with a focus on Wizard of Oz and developed systems Questionnaires on usability and user attitudes often used but few were reliable or validated Thematic analysis showed nine primary research topics Challenges identified in theoretical approaches and design guidelines, engaging with technological advances, multiple user and in the wild contexts, critical research mass and barriers to building speech interfaces
A New Mmwave-Speech Multimodal Speech System for Voice User Interface

Tiantian Liu,Feng Lin

DOI: https://doi.org/10.1145/3640087.3640096

2024-01-01

GetMobile Mobile Computing and Communications

Abstract:Voice user interface (VUI) plays an essential role in intelligent scenes, e.g., smart homes. It provides a hands- and eyes-free human-machine interaction between humans and Internet of Things devices. Benefiting from the development of deep learning and natural language process, the automatic speech recognition (ASR) entitles VUI to the capacity of accurate comprehension on users' intentions. With such a convenient and flexible service, users can interact with various devices as they please. Commercial VUI products have gained in popularity over recent years, such as smart speakers (e.g., Amazon Echo and Google Home), voice assistants in smartphones (e.g., Siri), and in-vehicle voice control interactions (e.g., VUIs in Tesla Model S/X/3/Y). Analysts forecast that by 2024, the deployment of VUI-based smart speakers will reach 640 million globally.
The Eighth Dialog System Technology Challenge

Kim Seokhwan,Galley Michel,Gunasekara Chulaka,Lee Sungjin,Atkinson Adam,Peng Baolin,Schulz Hannes,Gao Jianfeng,Li Jinchao,Adada Mahmoud,Huang Minlie,Lastras Luis,Kummerfeld Jonathan K.,Lasecki Walter S.,Hori Chiori,Cherian Anoop,Marks Tim K.,Rastogi Abhinav,Zang Xiaoxue,Sunkara Srinivas,Gupta Raghav

2019-01-01

Abstract: This paper introduces the Eighth Dialog System Technology Challenge. In line with recent challenges, the eighth edition focuses on applying end-to-end dialog technologies in a pragmatic way for multi-domain task-completion, noetic response selection, audio visual scene-aware dialog, and schema-guided dialog state tracking tasks. This paper describes the task definition, provided datasets, and evaluation set-up for each track. We also summarize the results of the submitted systems to highlight the overall trends of the state-of-the-art technologies for the tasks.
Special Issue on Mobile Big Data Management and Innovative Applications.

Kai Hwang,Min Chen,Jie Wu

DOI: https://doi.org/10.1109/tsc.2016.2601678

IF: 11.019

2016-01-01

IEEE Transactions on Services Computing

Abstract:The four papers in this special section aim to present high-quality contributions and innovations in this interdisciplinary area of mobile big data technologies, systems, and services, especially mobile big data management and innovative applications.
Medical Dialogue: A Survey of Categories, Methods, Evaluation and Challenges

Xiaoming Shi,Zeming Liu,Li Du,Yuxuan Wang,Hongru Wang,Yuhang Guo,Tong Ruan,Jie Xu,Shaoting Zhang

2024-05-17

Abstract:This paper surveys and organizes research works on medical dialog systems, which is an important yet challenging task. Although these systems have been surveyed in the medical community from an application perspective, a systematic review from a rigorous technical perspective has to date remained noticeably absent. As a result, an overview of the categories, methods, and evaluation of medical dialogue systems remain limited and underspecified, hindering the further improvement of this area. To fill this gap, we investigate an initial pool of 325 papers from well-known computer science, and natural language processing conferences and journals, and make an overview. Recently, large language models have shown strong model capacity on downstream tasks, which also reshaped medical dialog systems' foundation. Despite the alluring practical application value, current medical dialogue systems still suffer from problems. To this end, this paper lists the grand challenges of medical dialog systems, especially of large language models.

Computation and Language,Artificial Intelligence
Editorial for Special Issue “IberSPEECH2018: Speech and Language Technologies for Iberian Languages”

Francesc Alías,Antonio Bonafonte,António Teixeira

DOI: https://doi.org/10.3390/app10010384

2020-01-04

Applied Sciences

Abstract:The main goal of this Special Issue is to present the latest advances in research and novel applications of speech and language technologies based on the works presented at the IberSPEECH edition held in Barcelona in 2018, paying special attention to those focused on Iberian languages. IberSPEECH is the international conference of the Special Interest Group on Iberian Languages (SIG-IL) of the International Speech Communication Association (ISCA) and of the Spanish Thematic Network on Speech Technologies (Red Temática en Tecnologías del Habla, or RTTH for short). Several researchers were invited to extend their contributions presented at IberSPEECH2018 due to their interest and quality. As a result, this Special Issue is composed of 13 papers that cover different topics of investigation related to perception, speech analysis and enhancement, speaker verification and identification, speech production and synthesis, natural language processing, together with several applications and evaluation challenges.
Software-Based Dialogue Systems: Survey, Taxonomy and Challenges

Quim Motger,Xavier Franch,Jordi Marco

DOI: https://doi.org/10.1145/3527450

2024-02-06

Abstract:The use of natural language interfaces in the field of human-computer interaction is undergoing intense study through dedicated scientific and industrial research. The latest contributions in the field, including deep learning approaches like recurrent neural networks, the potential of context-aware strategies and user-centred design approaches, have brought back the attention of the community to software-based dialogue systems, generally known as conversational agents or chatbots. Nonetheless, and given the novelty of the field, a generic, context-independent overview on the current state of research of conversational agents covering all research perspectives involved is missing. Motivated by this context, this paper reports a survey of the current state of research of conversational agents through a systematic literature review of secondary studies. The conducted research is designed to develop an exhaustive perspective through a clear presentation of the aggregated knowledge published by recent literature within a variety of domains, research focuses and contexts. As a result, this research proposes a holistic taxonomy of the different dimensions involved in the conversational agents' field, which is expected to help researchers and to lay the groundwork for future research in the field of natural language interfaces.

Computation and Language,Software Engineering
Overview of the Eighth Dialog System Technology Challenge: DSTC8

Seokhwan Kim,Michel Galley,Chulaka Gunasekara,Sungjin Lee,Adam Atkinson,Baolin Peng,Hannes Schulz,Jianfeng Gao,Jinchao Li,Mahmoud Adada,Minlie Huang,Luis Lastras,Jonathan K. Kummerfeld,Walter S. Lasecki,Chiori Hori,Anoop Cherian,Tim K. Marks,Abhinav Rastogi,Xiaoxue Zang,Srinivas Sunkara,Raghav Gupta

DOI: https://doi.org/10.1109/taslp.2021.3078368

2021-01-01

Abstract:This paper introduces the Eighth Dialog System Technology Challenge. In line with recent challenges, the eighth edition focuses on applying end-to-end dialog technologies in a pragmatic way for multi-domain task-completion, noetic response selection, audio visual scene-aware dialog, and schema-guided dialog state tracking tasks. This paper describes the task definition, provided datasets, baselines and evaluation set-up for each track. We also summarize the results of the submitted systems to highlight the overall trends of the state-of-the-art technologies for the tasks.
Survey of evaluation methods for dialogue systems}{Survey of evaluation methods for dialogue systems

Wei-Nan Zhang,Yangzi Zhang,Ting Liu

DOI: https://doi.org/10.1360/n112017-00125

2017-01-01

Abstract:This paper introduces the history of dialogue systems and their evaluation methods. The evaluation methods are categorized as either task-oriented dialogue systems or open domain dialogue systems. This paper investigates and summarizes the different methods of evaluating dialogue systems, analyzes the pros and cons of the different methods, discusses the emphasis of each method, and presents the progress of recent research for the two categories. For task-oriented dialogue systems, this paper introduces the recent research results of Steve Young. In addition, this paper sums up several widely used evaluation approaches. The evaluation methods for open domain chatting systems are explored from two angles: objective index evaluation and simulated artificial scoring. The various indices and different research ideas are analyzed and introduced as well. Finally, through summarizing and analyzing classical evaluation methods of dialogue systems as well as the newer evaluation methods based on neural network models, this study aims to predict developmental trends in evaluation methods for dialogue systems.
Spoken language interaction with robots: Recommendations for future research

Matthew Marge,Carol Espy-Wilson,Nigel G. Ward,Abeer Alwan,Yoav Artzi,Mohit Bansal,Gil Blankenship,Joyce Chai,Hal Daumé,Debadeepta Dey,Mary Harper,Thomas Howard,Casey Kennington,Ivana Kruijff-Korbayová,Dinesh Manocha,Cynthia Matuszek,Ross Mead,Raymond Mooney,Roger K. Moore,Mari Ostendorf,Heather Pon-Barry,Alexander I. Rudnicky,Matthias Scheutz,Robert St. Amant,Tong Sun,Stefanie Tellex,David Traum,Zhou Yu

DOI: https://doi.org/10.1016/j.csl.2021.101255

2022-01-01

Abstract:<p>With robotics rapidly advancing, more effective human–robot interaction is increasingly needed to realize the full potential of robots for society. While spoken language must be part of the solution, our ability to provide spoken language interaction capabilities is still very limited. In this article, based on the report of an interdisciplinary workshop convened by the National Science Foundation, we identify key scientific and engineering advances needed to enable effective spoken language interaction with robotics. We make 25 recommendations, involving eight general themes: putting human needs first, better modeling the social and interactive aspects of language, improving robustness, creating new methods for rapid adaptation, better integrating speech and language with other communication modalities, giving speech and language components access to rich representations of the robot's current knowledge and state, making all components operate in real time, and improving research infrastructure and resources. Research and development that prioritizes these topics will, we believe, provide a solid foundation for the creation of speech-capable robots that are easy and effective for humans to work with.</p>

computer science, artificial intelligence
Dialogos: a robust system for human-machine spoken dialogue on the telephone

D. Albesano,P. Baggia,M. Danieli,R. Gemello,E. Gerbino,C. Rullent

DOI: https://doi.org/10.1109/icassp.1997.596145

2024-01-08

Abstract:This paper presents Dialogos, a real-time system for human-machine spoken dialogue on the telephone in task-oriented domains. The system has been tested in a large trial with inexperienced users and it has proved robust enough to allow spontaneous interactions both to users which get good recognition performance and to the ones which get lower scores. The robust behavior of the system has been achieved by combining the use of specific language models during the recognition phase of analysis, the tolerance toward spontaneous speech phenomena, the activity of a robust parser, and the use of pragmatic-based dialogue knowledge. This integration of the different modules allows to deal with partial or total breakdowns of the different levels of analysis. We report the field trial data of the system and the evaluation results of the overall system and of the submodules.

Introduction to the Issue on Advances in Spoken Dialogue Systems and Mobile Interface

Editorial: Recent advances in intelligent robotic systems

Special issue on multimodal processing and robotics for dialogue systems (Part II)

Teaching Machines to Converse

A Summary of Research on Intelligent Dialogue Systems

Dialogue with Robots: Proposals for Broadening Participation and Research in the SLIVAR Community

WavChat: A Survey of Spoken Dialogue Models

Talking with Machines: A Comprehensive Survey of Emergent Dialogue Systems

Recent Advances and Challenges in Task-Oriented Dialog Systems

Challenges in Building Intelligent Open-domain Dialog Systems

The State of Speech in HCI: Trends, Themes and Challenges

A New Mmwave-Speech Multimodal Speech System for Voice User Interface

The Eighth Dialog System Technology Challenge

Special Issue on Mobile Big Data Management and Innovative Applications.

Medical Dialogue: A Survey of Categories, Methods, Evaluation and Challenges

Editorial for Special Issue “IberSPEECH2018: Speech and Language Technologies for Iberian Languages”

Software-Based Dialogue Systems: Survey, Taxonomy and Challenges

Overview of the Eighth Dialog System Technology Challenge: DSTC8

Survey of evaluation methods for dialogue systems}{Survey of evaluation methods for dialogue systems

Spoken language interaction with robots: Recommendations for future research

Dialogos: a robust system for human-machine spoken dialogue on the telephone