Abstract:Speech interaction systems have been gaining popularity in recent years. The main purpose of these systems is to generate more satisfactory responses according to users' speech utterances, in which the most critical problem is to analyze user intention. Researches show that user intention conveyed through speech is not only expressed by content, but also closely related with users' speaking manners (e.g. with or without acoustic emphasis). How to incorporate these heterogeneous attributes to infer user intention remains an open problem. In this paper, we define Intention Prominence (IP) as the semantic combination of focus by text and emphasis by speech, and propose a multi-task deep learning framework to predict IP. Specifically, we first use long short-term memory (LSTM) which is capable of modeling long short-term contextual dependencies to detect focus and emphasis, and incorporate the tasks for focus and emphasis detection with multi-task learning (MTL) to reinforce the performance of each other. We then employ Bayesian network (BN) to incorporate multimodal features (focus, emphasis, and location reflecting users' dialect conventions) to predict IP based on feature correlations. Experiments on a data set of 135,566 utterances collected from real-world Sogou Voice Assistant illustrate that our method can outperform the comparison methods over 6.9-24.5% in terms of F1-measure. Moreover, a real practice in the Sogou Voice Assistant indicates that our method can improve the performance on user intention understanding by 7%.

Multilingual and Cross-Lingual Intent Detection from Spoken Data

Multi-Lingual Attention based Multi-Intent Detection in Dialogue System

Real-time Caller Intent Detection In Human-Human Customer Support Spoken Conversations

Investigation of Multilingual Deep Neural Networks for Spoken Term Detection.

Transferring Audio Deepfake Detection Capability Across Languages

Acoustics Based Intent Recognition Using Discovered Phonetic Units for Low Resource Languages

Attention-Informed Mixed-Language Training for Zero-Shot Cross-Lingual Task-Oriented Dialogue Systems

Intent Detection in the Age of LLMs

Zero-Shot Slot and Intent Detection in Low-Resource Languages

Multi-Task Deep Learning for User Intention Understanding in Speech Interaction Systems

An Intent Recognition Model Supporting the Spoken Expression Mixed with Chinese and English.

Intent Recognition and Unsupervised Slot Identification for Low Resourced Spoken Dialog Systems

Listen with Intent: Improving Speech Recognition with Audio-to-Intent Front-End

Investigating cross-lingual training for offensive language detection

Leveraging Acoustic and Linguistic Embeddings from Pretrained speech and language Models for Intent Classification

Intent Classification Using Pre-trained Language Agnostic Embeddings For Low Resource Languages

CLID: A Chunk-Level Intent Detection Framework for Multiple Intent Spoken Language Understanding.

Heterogeneity over Homogeneity: Investigating Multilingual Speech Pre-Trained Models for Detecting Audio Deepfake

Multimodal Data and Resource Efficient Device-Directed Speech Detection with Large Foundation Models

A Multimodal Approach to Device-Directed Speech Detection with Large Language Models

Speech Activity Detection Based on Multilingual Speech Recognition System