Abstract:Subtitles are a crucial component of Digital Entertainment Content (DEC such as movies and TV shows) localization. With ever increasing catalog (≈ 2M titles) and localization expansion (30+ languages), automated subtitle quality checks becomes paramount. Being a manual creation process, subtitles can have errors such as missing transcriptions, out-of-sync subtitle blocks with the audio and incorrect translations. Such erroneous subtitles result in an unpleasant viewing experience and impact the viewership. Moreover, manual correction is laborious, highly costly and requires expertise of audio and subtitle languages. A typical subtitle correction process consists of (1) linear watch of the movie, (2) identification of time stamps associated with erroneous subtitle blocks, and (3) correcting procedure. Among the three, time taken to watch the entire movie by a human expert is the most time consuming step. This paper discusses the problem of missing transcription, where the subtitle blocks corresponding to some speech segments in the DEC are non-existent. We present a solution to augment human correction process by automatically identifying the timings associated with the non-transcribed dialogues in a language agnostic manner. The correction step can then be performed by either human-in-the-loop mechanism or automatically using neural transcription (speech-to-text in same language) and translation (text-to-text in different languages) engines. Our method uses a language agnostic neural voice activity detector (VAD) and an audio classifier (AC) trained explicitly on DEC corpora for better generalization. The method consists of three steps: first, we use VAD to identify the timings associated with dialogues (predicted speech blocks). Second, we refine those timings using the AC module by removing the timings associated with the leading and trailing non-speech segments identified as speech by VAD. Finally, we compare the predicted dialogue timings to the dialogue timings present in the subtitle file (subtitle speech blocks) and flag the missing transcriptions. We empirically demonstrate that the proposed method (a) reduces incorrect predicted missing subtitle timings by 10%, (b) improves the predicted missing subtitle timings by 2.5%, (c) reduces false positive rate (FPR) of overextending the predicted timings by 77%, and (d) improves the predicted speech block-level precision by a 119% over VAD baseline on a human-annotated dataset of missing subtitle speech blocks.

A Survey Study on Automatic Subtitle Synchronization and Positioning System for Deaf and Hearing Impaired People

Caption positioning structure for hard of hearing people using deep learning method

Language agnostic missing subtitle detection

Video accessibility enhancement for hearing-impaired users

Joint Generation of Captions and Subtitles with Dual Decoding

Video Content Aware Dynamic Subtitles

Speaker-following Video Subtitles

Multilingual Video Viewing Subtitle to Audio Translator

Character-aware audio-visual subtitling in context

SBAAM! Eliminating Transcript Dependency in Automatic Subtitling

Look, Listen and Recognise: Character-Aware Audio-Visual Subtitling

Automated 3D sign language caption generation for video

Video captioning: a review of theory, techniques and practices

Aligning Subtitles in Sign Language Videos

All Country Sign Language Recognition System to Help Deaf and Mute People

A subjective study of the perceptual acceptability of audio-video desynchronization in sports videos

A quantitative analysis of the reactions of viewers with hearing impairment to the intralingual subtitling of Egyptian movies

Video captioning – a survey

Video2Subtitle: Matching Weakly-Synchronized Sequences Via Dynamic Temporal Alignment

Investigations in Audio Captioning: Addressing Vocabulary Imbalance and Evaluating Suitability of Language-Centric Performance Metrics

A Simple and Efficient method for Dubbed Audio Sync Detection using Compressive Sensing