Abstract:Robust face clustering is a vital step in enabling computational understanding of visual character portrayal in media. Face clustering for long-form content is challenging because of variations in appearance and lack of supporting large-scale labeled data. Our work in this paper focuses on two key aspects of this problem: the lack of domain-specific training or benchmark datasets, and adapting face embeddings learned on web images to long-form content, specifically movies. First, we present a dataset of over 169,000 face tracks curated from 240 Hollywood movies with weak labels on whether a pair of face tracks belong to the same or a different character. We propose an offline algorithm based on nearest-neighbor search in the embedding space to mine hard-examples from these tracks. We then investigate triplet-loss and multiview correlation-based methods for adapting face embeddings to hard-examples. Our experimental results highlight the usefulness of weakly labeled data for domain-specific feature adaptation. Overall, we find that multiview correlation-based adaptation yields more discriminative and robust face embeddings. Its performance on downstream face verification and clustering tasks is comparable to that of the state-of-the-art results in this domain. We also present the SAIL-Movie Character Benchmark corpus developed to augment existing benchmarks. It consists of racially diverse actors and provides face-quality labels for subsequent error analysis. We hope that the large-scale datasets developed in this work can further advance automatic character labeling in videos. All resources are available freely at <a class="link-external link-https" href="https://sail.usc.edu/~ccmi/multiface" rel="external noopener nofollow">this https URL</a>.

“Who are you?” - Learning person specific classifiers from video

Automated Video Labelling: Identifying Faces by Corroborative Evidence

From Benedict Cumberbatch to Sherlock Holmes: Character Identification in TV series without a Script

Who's that Actor? Automatic Labelling of Actors in TV series starting from IMDB Images

Automatic Naming of Speakers in Video via Name-Face Mapping.

Character-aware audio-visual subtitling in context

Cast2Face

Robust Character Labeling in Movie Videos: Data Resources and Self-supervised Feature Adaptation

Look, Listen and Recognise: Character-Aware Audio-Visual Subtitling

Audio visual character profiles for detecting background characters in entertainment media

Face identification using reference-based features with message passing model

Adaptive Learning for Celebrity Identification with Video Context

Character Matters: Video Story Understanding with Character-Aware Relations

Long-term face tracking in the wild using deep learning

Learnable PINs: Cross-Modal Embeddings for Person Identity

VideoClusterNet: Self-Supervised and Adaptive Face Clustering For Videos

Incorporating Generic Learning to Design Discriminative Classifier Adaptable for Unknown Subject in Face Verification.

Playing a Part: Speaker Verification at the Movies

Learning Person-specific Cognition from Facial Reactions for Automatic Personality Recognition

Multiple instance learning for labeling faces in broadcasting news video

Learning Grimaces by Watching TV