Abstract:Data simulation engines like Unity are becoming an increasingly important data source that allows us to acquire ground truth labels conveniently. Moreover, we can flexibly edit the content of an image in the engine, such as objects (position, orientation) and environments (illumination, occlusion). When using simulated data as training sets, its editable content can be leveraged to mimick the distribution of real-world data, and thus reduce the content difference between the synthetic and real domains. This paper explores content adaptation in the context of semantic segmentation, where the complex street scenes are fully synthesized using 19 classes of virtual objects from a first person driver perspective and controlled by 23 attributes. To optimize the attribute values and obtain a training set of similar content to real-world data, we propose a scalable discretization-and-relaxation (SDR) approach. Under a reinforcement learning framework, we formulate attribute optimization as a random-to-optimized mapping problem using a neural network. Our method has three characteristics. 1) Instead of editing attributes of individual objects, we focus on global attributes that have large influence on the scene structure, such as object density and illumination. 2) Attributes are quantized to discrete values, so as to reduce search space and training complexity. 3) Correlated attributes are jointly optimized in a group, so as to avoid meaningless scene structures and find better convergence points. Experiment shows our system can generate reasonable and useful scenes, from which we obtain promising real-world segmentation accuracy compared with existing synthetic training sets.

FairScene: Learning Unbiased Object Interactions for Indoor Scene Synthesis

Learning 3 D Scene Synthesis from Annotated RGB-D Images

Learning to Simulate Complex Scenes for Street Scene Segmentation

PCPL: Predicate-Correlation Perception Learning for Unbiased Scene Graph Generation

S-INF: Towards Realistic Indoor Scene Synthesis via Scene Implicit Neural Field

Fast 3D Indoor Scene Synthesis by Learning Spatial Relation Priors of Objects

Fast 3D Indoor Scene Synthesis by LearningSpatial Relation Priors of Objects

Biasing Like Human: A Cognitive Bias Framework for Scene Graph Generation

Unbiased Scene Graph Generation via Rich and Fair Semantic Extraction

Fast Contextual Scene Graph Generation with Unbiased Context Augmentation.

Logical Bias Learning for Object Relation Prediction

Neural Rendering in a Room: Amodal 3D Understanding and Free-Viewpoint Rendering for the Closed Scene Composed of Pre-Captured Objects

Debiased Scene Graph Generation for Dual Imbalance Learning

CLIP-Layout: Style-Consistent Indoor Scene Synthesis with Semantic Furniture Embedding

DiffuScene: Denoising Diffusion Models for Generative Indoor Scene Synthesis

Planner3D: LLM-enhanced graph prior meets 3D indoor scene explicit regularization

Active Arrangement of Small Objects in 3D Indoor Scenes

Fast 3D Indoor Scene Synthesis with Discrete and Exact Layout Pattern Extraction

Adaptive synthesis of indoor scenes via activity-associated object relation graphs.

IFR-Explore: Learning Inter-object Functional Relationships in 3D Indoor Scenes

Explicit3D: Graph Network with Spatial Inference for Single Image 3D Object Detection