Abstract:Recent voice assistants are usually based on the cascade spoken language understanding (SLU) solution, which consists of an automatic speech recognition (ASR) engine and a natural language understanding (NLU) system. Because such approach relies on the ASR output, it often suffers from the so-called ASR error propagation. In this work, we investigate impacts of this ASR error propagation on state-of-the-art NLU systems based on pre-trained language models (PLM), such as BERT and RoBERTa. Moreover, a multimodal language understanding (MLU) module is proposed to mitigate SLU performance degradation caused by errors present in the ASR transcript. The MLU benefits from self-supervised features learned from both audio and text modalities, specifically Wav2Vec for speech and Bert/RoBERTa for language. Our MLU combines an encoder network to embed the audio signal and a text encoder to process text transcripts followed by a late fusion layer to fuse audio and text logits. We found that the proposed MLU showed to be robust towards poor quality ASR transcripts, while the performance of BERT and RoBERTa are severely compromised. Our model is evaluated on five tasks from three SLU datasets and robustness is tested using ASR transcripts from three ASR engines. Results show that the proposed approach effectively mitigates the ASR error propagation problem, surpassing the PLM models' performance across all datasets for the academic ASR engine.

Robust Language Understanding Algorithm in Spoken Dialogue Systems

Robust Voice Command Understanding and Error Tolerance Algorithm Based on Word Graph Expansion

A New Robust and Domain-Oriented Algorithm of Text Parsing

Robust Parsing in Spoken Dialogue Systems.

ASR-Robust Spoken Language Understanding on ASR-GLUE dataset

Dialogos: a robust system for human-machine spoken dialogue on the telephone

Mandarin Language Understanding in Dialogue Context

An Adversarial Learning based Multi-Step Spoken Language Understanding System through Human-Computer Interaction

An effective and robust approach to Mandarin spoken language understanding in specific domain

Robust Speech Recognition Directed by Extended Template Matching in Dialogue System

Robustness Testing of Language Understanding in Task-Oriented Dialog

Language Understanding Component for Chinese Dialogue System.

Towards ASR Robust Spoken Language Understanding Through In-Context Learning With Word Confusion Networks

Jointly Encoding Word Confusion Network and Dialogue Context with BERT for Spoken Language Understanding

Multimodal Audio-textual Architecture for Robust Spoken Language Understanding

DialogVCS: Robust Natural Language Understanding in Dialogue System Upgrade

Socaog: Incremental Graph Parsing For Social Relation Inference In Dialogues

Improving Robustness of Task Oriented Dialog Systems

Towards Robust Online Dialogue Response Generation

Non-Fuchsian extension to the Painlevé test

ROBUSTNESS OF SPEECH RECOGNITION AND CONSTRUCTION OF A SPEECH CORPUS