Abstract:Word vector representations are a crucial part of natural language processing (NLP) and human computer interaction. In this paper, we propose a novel word vector representation, Confusion2Vec, motivated from the human speech production and perception that encodes representational ambiguity. Humans employ both acoustic similarity cues and contextual cues to decode information and we focus on a model that incorporates both sources of information. The representational ambiguity of acoustics, which manifests itself in word confusions, is often resolved by both humans and machines through contextual cues. A range of representational ambiguities can emerge in various domains further to acoustic perception, such as morphological transformations, word segmentation, paraphrasing for NLP tasks like machine translation, etc. In this work, we present a case study in application to automatic speech recognition (ASR) task, where the word representational ambiguities/confusions are related to acoustic similarity. We present several techniques to train an acoustic perceptual similarity representation ambiguity. We term this Confusion2Vec and learn on unsupervised-generated data from ASR confusion networks or lattice-like structures. Appropriate evaluations for the Confusion2Vec are formulated for gauging acoustic similarity in addition to semantic–syntactic and word similarity evaluations. The Confusion2Vec is able to model word confusions efficiently, without compromising on the semantic-syntactic word relations, thus effectively enriching the word vector space with extra task relevant ambiguity information. We provide an intuitive exploration of the two-dimensional Confusion2Vec space using principal component analysis of the embedding and relate to semantic relationships, syntactic relationships, and acoustic relationships. We show through this that the new space preserves the semantic/syntactic relationships while robustly encoding acoustic similarities. The potential of the new vector representation and its ability in the utilization of uncertainty information associated with the lattice is demonstrated through small examples relating to the task of ASR error correction.

Confides: A Visual Analytics Solution for Automated Speech Recognition Analysis and Exploration

Confidence-features and Confidence-Scores for ASR Applications in Arbitration and DNN Speaker Adaptation.

Using Confidence Scores to Improve Eyes-free Detection of Speech Recognition Errors

The use of artificial intelligence in interrogations: voluntary confession

CONFIDERAI: a novel CONFormal Interpretable-by-Design score function for Explainable and Reliable Artificial Intelligence

Designing for Confidence: The Impact of Visualizing Artificial Intelligence Decisions

SpeechLens: A Visual Analytics Approach for Exploring Speech Strategies with Textural and Acoustic Features

Investigating Confidence Estimation Measures for Speaker Diarization

A Pilot Study on Clinician-AI Collaboration in Diagnosing Depression from Speech

Lost in Transcription: Identifying and Quantifying the Accuracy Biases of Automatic Speech Recognition Systems Against Disfluent Speech

Confidence Estimation for Automatic Detection of Depression and Alzheimer's Disease Based on Clinical Interviews

UNCOVER: Identifying AI Generated News Articles by Linguistic Analysis and Visualization

Investigating Trust in Human-AI Collaboration for a Speech-Based Data Analytics Task

Towards Better Understanding of Spontaneous Conversations: Overcoming Automatic Speech Recognition Errors With Intent Recognition

Unveiling hidden factors: explainable AI for feature boosting in speech emotion recognition

Accurate and Reliable Confidence Estimation Based on Non-Autoregressive End-to-End Speech Recognition System

Reliable Visualization for Deep Speaker Recognition

Task Oriented Dialogue as a Catalyst for Self-Supervised Automatic Speech Recognition

Confusion2Vec: towards enriching vector space word representations with representational ambiguities

Sequence-level Confidence Classifier for ASR Utterance Accuracy and Application to Acoustic Models

Confident AI