Abstract:Scene graph generation (SGG) analyzes images to extract meaningful information about objects and their relationships. In the dynamic visual world, it is crucial for AI systems to continuously detect new objects and establish their relationships with existing ones. Recently, numerous studies have focused on continual learning within the domains of object detection and image recognition. However, a limited amount of research focuses on a more challenging continual learning problem in SGG. This increased difficulty arises from the intricate interactions and dynamic relationships among objects, and their associated contexts. Thus, in continual learning, SGG models are often required to expand, modify, retain, and reason scene graphs within the process of adaptive visual scene understanding. To systematically explore Continual Scene Graph Generation (CSEGG), we present a comprehensive benchmark comprising three learning regimes: relationship incremental, scene incremental, and relationship generalization. Moreover, we introduce a ``Replays via Analysis by Synthesis" method named RAS. This approach leverages the scene graphs, decomposes and re-composes them to represent different scenes, and replays the synthesized scenes based on these compositional scene graphs. The replayed synthesized scenes act as a means to practice and refine proficiency in SGG in known and unknown environments. Our experimental results not only highlight the challenges of directly combining existing continual learning methods with SGG backbones but also demonstrate the effectiveness of our proposed approach, enhancing CSEGG efficiency while simultaneously preserving privacy and memory usage. All data and source code are publicly available online.

What problem does this paper attempt to address?

The problem that this paper attempts to solve is how to enable artificial intelligence systems to continuously detect new objects and establish relationships between them and existing objects in dynamic visual scenes, especially in the Continual Learning in Scene Graph Generation (SGG) tasks. Specifically, the paper focuses on the **Continual Scene Graph Generation (CSEGG)** problem. This problem is more complex than the Continual Learning in traditional object detection and image recognition because it needs to handle the complex interactions and dynamic relationships between objects, as well as the changes of these relationships over time. ### Main Challenges 1. **Complex Object Relationship Understanding**: Unlike object detection, SGG not only needs to identify objects but also understand the relationships between them, which becomes more complex in dynamic scenes. 2. **Combinatorial Complexity**: Each detected pair of objects may have multiple potential spatial and functional relationships. Therefore, as new objects are introduced, the relationship complexity between objects will increase significantly in a non - linear manner. 3. **Long - Tail Distribution**: Scenes in the real world have the long - tail distribution characteristic, that is, some objects are more common than others. Therefore, CSEGG needs to continuously adapt to this changing long - tail distribution. ### Solutions To address the above challenges, the paper proposes a method named "Replays via Analysis by Synthesis (RAS)". The main features of RAS include: 1. **Symbolic Replay**: RAS utilizes the scene graphs in previous tasks, decomposes and recombines them to generate diverse scene structures. These combined scene graphs are used to synthesize scene images for replay. 2. **Privacy Protection and Data Efficiency**: Since RAS uses symbolic replay, it does not need to store the original images, thus ensuring data privacy and storage efficiency. 3. **Semantic Context Preservation**: By synthesizing scene images, RAS not only preserves the semantic context and structure of previous scenes but also enhances the diversity of scene generation. 4. **Balancing Long - Tail Distribution**: RAS ensures the uniform sampling of objects and relationships during the replay process by balancing the distribution of tail and head categories, preventing prediction bias. ### Experimental Results The paper conducted extensive experiments through three learning scenarios (relational incremental learning, scene incremental learning, and relational generalization). The results show that when existing Continual Learning methods are directly applied to SGG, the effect is not good, while the RAS method can not only improve the efficiency of CSEGG but also effectively prevent forgetting and maintain good performance in both known and unknown environments. ### Summary By proposing the RAS method, this paper systematically explores the CSEGG problem and provides a new benchmark and methodological basis for future research. The RAS method not only technically solves the challenges of CSEGG but also shows its potential in practical applications, such as real - time robot navigation and adaptive augmented reality experiences.

Adaptive Visual Scene Understanding: Incremental Scene Graph Generation

Fast Contextual Scene Graph Generation with Unbiased Context Augmentation.

Continuous Self-study: Scene Graph Generation with Self-knowledge Distillation and Spatial Augmentation

Towards Lifelong Scene Graph Generation with Knowledge-ware In-context Prompt Learning

Attention Redirection Transformer with Semantic Oriented Learning for Unbiased Scene Graph Generation

Scene Graph Modification as Incremental Structure Expanding

Scene Dynamics: Counterfactual Critic Multi-Agent Training for Scene Graph Generation.

Learning to Generate Scene Graph from Head to Tail

Scene Graph Generation: A Comprehensive Survey

Relationship-Incremental Scene Graph Generation by a Divide-and-Conquer Pipeline with Feature Adapter

Learning Canonical Representations for Scene Graph to Image Generation

Improving Scene Graph Generation with Superpixel-Based Interaction Learning

Fine-Grained is Too Coarse: A Novel Data-Centric Approach for Efficient Scene Graph Generation

SGEITL: Scene Graph Enhanced Image-Text Learning for Visual Commonsense Reasoning

Beware of Overcorrection: Scene-induced Commonsense Graph for Scene Graph Generation

HiKER-SGG: Hierarchical Knowledge Enhanced Robust Scene Graph Generation

From General to Specific: Informative Scene Graph Generation Via Balance Adjustment

Embodied Semantic Scene Graph Generation.

Learning to Generate Language-Supervised and Open-Vocabulary Scene Graph Using Pre-Trained Visual-Semantic Space

A Comprehensive Survey of Scene Graphs: Generation and Application

Scene Graph Generation with Role-Playing Large Language Models