Abstract:This paper investigates the problem of scene graph generation in videos with the aim of capturing semantic relations between subjects and objects in the form of $\langle$subject, predicate, object$\rangle$ triplets. Recognizing the predicate between subject and object pairs is imbalanced and multi-label in nature, ranging from ubiquitous interactions such as spatial relationships (\eg \emph{in front of}) to rare interactions such as \emph{twisting}. In widely-used benchmarks such as Action Genome and VidOR, the imbalance ratio between the most and least frequent predicates reaches 3,218 and 3,408, respectively, surpassing even benchmarks specifically designed for long-tailed recognition. Due to the long-tailed distributions and label co-occurrences, recent state-of-the-art methods predominantly focus on the most frequently occurring predicate classes, ignoring those in the long tail. In this paper, we analyze the limitations of current approaches for scene graph generation in videos and identify a one-to-one correspondence between predicate frequency and recall performance. To make the step towards unbiased scene graph generation in videos, we introduce a multi-label meta-learning framework to deal with the biased predicate distribution. Our meta-learning framework learns a meta-weight network for each training sample over all possible label losses. We evaluate our approach on the Action Genome and VidOR benchmarks by building upon two current state-of-the-art methods for each benchmark. The experiments demonstrate that the multi-label meta-weight network improves the performance for predicates in the long tail without compromising performance for head classes, resulting in better overall performance and favorable generalizability. Code: \url{<a class="link-external link-https" href="https://github.com/shanshuo/ML-MWN" rel="external noopener nofollow">this https URL</a>}.

Bel: Batch Equalization Loss for Scene Graph Generation

PCPL: Predicate-Correlation Perception Learning for Unbiased Scene Graph Generation

From General to Specific: Informative Scene Graph Generation Via Balance Adjustment

PPDL: Predicate Probability Distribution Based Loss for Unbiased Scene Graph Generation

The Equalization Losses: Gradient-Driven Training for Long-tailed Object Recognition

Do We Really Reduce Bias for Scene Graph Generation?

Attention Redirection Transformer with Semantic Oriented Learning for Unbiased Scene Graph Generation

Fine-Grained Scene Graph Generation via Sample-Level Bias Prediction

Dark Knowledge Balance Learning for Unbiased Scene Graph Generation

An Effective Dynamic Reweighting Method for Unbiased Scene Graph Generation.

A Causality Guided Loss for Imbalanced Learning in Scene Graph Generation

A New Training Data Organization Form and Training Mode for Unbiased Scene Graph Generation

Resistance Training Using Prior Bias: Toward Unbiased Scene Graph Generation

Equalization Loss V2: A New Gradient Balance Approach for Long-tailed Object Detection

Fast Contextual Scene Graph Generation with Unbiased Context Augmentation.

Debiased Scene Graph Generation for Dual Imbalance Learning

Informative Scene Graph Generation via Debiasing

Stacked Hybrid-Attention and Group Collaborative Learning for Unbiased Scene Graph Generation

Learning from the Scene and Borrowing from the Rich: Tackling the Long Tail in Scene Graph Generation

A Hybrid Loss Balancing Algorithm Based on Gradient Equilibrium and Sample Loss for Understanding of Road Scenes at Basic-Level

Multi-Label Meta Weighting for Long-Tailed Dynamic Scene Graph Generation