Abstract:Spoken content retrieval refers to directly indexing and retrieving spoken content based on the audio rather than text descriptions. This potentially eliminates the requirement of producing text descriptions for multimedia content for indexing and retrieval purposes, and is able to precisely locate the exact time the desired information appears in the multimedia. Spoken content retrieval has been very successfully achieved with the basic approach of cascading automatic speech recognition (ASR) with text information retrieval: after the spoken content is transcribed into text or lattice format, a text retrieval engine searches over the ASR output to find desired information. This framework works well when the ASR accuracy is relatively high, but becomes less adequate when more challenging real-world scenarios are considered, since retrieval performance depends heavily on ASR accuracy. This challenge leads to the emergence of another approach to spoken content retrieval: to go beyond the basic framework of cascading ASR with text retrieval in order to have retrieval performances that are less dependent on ASR accuracy. This overview article is intended to provide a thorough overview of the concepts, principles, approaches, and achievements of major technical contributions along this line of investigation. This includes five major directions: 1) Modified ASR for Retrieval Purposes: cascading ASR with text retrieval, but the ASR is modified or optimized for spoken content retrieval purposes; 2) Exploiting the Information not present in ASR outputs: to try to utilize the information in speech signals inevitably lost when transcribed into phonemes and words; 3) Directly Matching at the Acoustic Level without ASR: for spoken queries, the signals can be directly matched at the acoustic level, rather than at the phoneme or word levels, bypassing all ASR issues; 4) Semantic Retrieval of Spoken Content: trying to retrieve spoken content that is semanti- ally related to the query, but not necessarily including the query terms themselves; 5) Interactive Retrieval and Efficient Presentation of the Retrieved Objects: with efficient presentation of the retrieved objects, an interactive retrieval process incorporating user actions may produce better retrieval results and user experiences.

Enhancing Query Expansion for Semantic Retrieval of Spoken Content with Automatically Discovered Acoustic Patterns.

Towards Unsupervised Semantic Retrieval Of Spoken Content With Query Expansion Based On Automatically Discovered Acoustic Patterns

Improved Semantic Retrieval of Spoken Content by Document/Query Expansion with Random Walk Over Acoustic Similarity Graphs

Improved Semantic Retrieval of Spoken Content by Language Models Enhanced with Acoustic Similarity Graph

Query Expansion Based High Performance Chinese Voice Retrieval

A Semantic Query Expansion-Based Patent Retrieval Approach

Semantic Query Expansion and Context-Based Discriminative Term Modeling for Spoken Document Retrieval

Semantic-based Sound Retrieval by ERP in Rapid Serial Auditory Presentation Paradigm.

Quality-aware Aggregated Conformal Prediction for Silent Speech Recognition

Spoken Content Retrieval—Beyond Cascading Speech Recognition with Text Retrieval

Open-Vocabulary Retrieval of Spoken Content with Shorter/Longer Queries Considering Word/Subword-based Acoustic Feature Similarity.

High-precision Voice Search Query Correction via Retrievable Speech-text Embedings

Improved open-vocabulary spoken content retrieval with word and subword lattices using acoustic feature similarity

Expand, Rerank, and Retrieve: Query Reranking for Open-Domain Question Answering.

Phonetic-and-Semantic Embedding of Spoken Words with Applications in Spoken Content Retrieval

Improved Spoken Term Detection by Discriminative Training of Acoustic Models Based on User Relevance Feedback.

A Revisit Of Query Expansion With Different Semantic Levels

AVATAR: Robust Voice Search Engine Leveraging Autoregressive Document Retrieval and Contrastive Learning

Improving keyword search by query expansion in a probabilistic framework

Scalable Identity-Oriented Speech Retrieval

Unsupervised Discovery of Structured Acoustic Tokens with Applications to Spoken Term Detection