Abstract:Subtitles are a crucial component of Digital Entertainment Content (DEC such as movies and TV shows) localization. With ever increasing catalog (≈ 2M titles) and localization expansion (30+ languages), automated subtitle quality checks becomes paramount. Being a manual creation process, subtitles can have errors such as missing transcriptions, out-of-sync subtitle blocks with the audio and incorrect translations. Such erroneous subtitles result in an unpleasant viewing experience and impact the viewership. Moreover, manual correction is laborious, highly costly and requires expertise of audio and subtitle languages. A typical subtitle correction process consists of (1) linear watch of the movie, (2) identification of time stamps associated with erroneous subtitle blocks, and (3) correcting procedure. Among the three, time taken to watch the entire movie by a human expert is the most time consuming step. This paper discusses the problem of missing transcription, where the subtitle blocks corresponding to some speech segments in the DEC are non-existent. We present a solution to augment human correction process by automatically identifying the timings associated with the non-transcribed dialogues in a language agnostic manner. The correction step can then be performed by either human-in-the-loop mechanism or automatically using neural transcription (speech-to-text in same language) and translation (text-to-text in different languages) engines. Our method uses a language agnostic neural voice activity detector (VAD) and an audio classifier (AC) trained explicitly on DEC corpora for better generalization. The method consists of three steps: first, we use VAD to identify the timings associated with dialogues (predicted speech blocks). Second, we refine those timings using the AC module by removing the timings associated with the leading and trailing non-speech segments identified as speech by VAD. Finally, we compare the predicted dialogue timings to the dialogue timings present in the subtitle file (subtitle speech blocks) and flag the missing transcriptions. We empirically demonstrate that the proposed method (a) reduces incorrect predicted missing subtitle timings by 10%, (b) improves the predicted missing subtitle timings by 2.5%, (c) reduces false positive rate (FPR) of overextending the predicted timings by 77%, and (d) improves the predicted speech block-level precision by a 119% over VAD baseline on a human-annotated dataset of missing subtitle speech blocks.

Detecting False Alarms and Misses in Audio Captions

Resource-Efficient Reference-Free Evaluation of Audio Captions

Investigations in Audio Captioning: Addressing Vocabulary Imbalance and Evaluating Suitability of Language-Centric Performance Metrics

EMScore: Evaluating Video Captioning via Coarse-Grained and Fine-Grained Embedding Matching

Language agnostic missing subtitle detection

Cobra Effect in Reference-Free Image Captioning Metrics

Measuring the Accuracy of Automatic Speech Recognition Solutions

Synthesizing Speech Test Cases with Text-to-Speech? An Empirical Study on the False Alarms in Automated Speech Recognition Testing

ACES: Evaluating Automated Audio Captioning Models on the Semantics of Sounds

An investigation on selecting audio pre-trained models for audio captioning

Soundscape Captioning using Sound Affective Quality Network and Large Language Model

Caption-aided Speech Detection in Videos

Zero-Shot Audio Captioning via Audibility Guidance

Seeing and Hearing Too: Audio Representation for Video Captioning.

Learning to Evaluate Image Captioning

Models See Hallucinations: Evaluating the Factuality in Video Captioning

MACE: Leveraging Audio for Evaluating Audio Captioning Systems

Quality-agnostic Image Captioning to Safely Assist People with Vision Impairment

Misinformation Detection on YouTube Using Video Captions

Exploration of Audio Quality Assessment and Anomaly Localisation Using Attention Models

ALCAP: Alignment-Augmented Music Captioner