Abstract:Generating virtual organ populations that capture sufficient variability while remaining plausible is essential to conduct in silico trials (ISTs) of medical devices. However, not all anatomical shapes of interest are always available for each individual in a population. The imaging examinations and modalities used can vary between subjects depending on their individualized clinical pathways. Different imaging modalities may have various fields of view and are sensitive to signals from other tissues/organs, or both. Hence, missing/partially overlapping anatomical information is often available across individuals. We introduce a generative shape model for multipart anatomical structures, learnable from sets of unpaired datasets, i.e., where each substructure in the shape assembly comes from datasets with missing or partially overlapping substructures from disjoint subjects of the same population. The proposed generative model can synthesize complete multipart shape assemblies coined virtual chimeras (VCs). We applied this framework to build VCs from databases of whole-heart shape assemblies that each contribute samples for heart substructures. Specifically, we propose a graph neural network-based generative shape compositional framework, which comprises two components, a part-aware generative shape model that captures the variability in shape observed for each structure of interest in the training population and a spatial composition network that assembles/composes the structures synthesized by the former into multipart shape assemblies (i.e., VCs). We also propose a novel self-supervised learning scheme that enables the spatial composition network to be trained with partially overlapping data and weak labels. We trained and validated our approach using shapes of cardiac structures derived from cardiac magnetic resonance (MR) images in the UK Biobank (UKBB). When trained with complete and partially overlapping data, our approach significantly outperforms a principal component analysis (PCA)-based shape model (trained with complete data) in terms of generalizability and specificity. This demonstrates the superiority of the proposed method, as the synthesized cardiac virtual populations are more plausible and capture a greater degree of shape variability than those generated by the PCA-based shape model.

Progressive Multimodal Shape Generation via Contextual Part Reasoning

Diverse Part Synthesis for 3D Shape Creation

Multimodal Shape Completion Via Conditional Generative Adversarial Networks

PT2PC: Learning to Generate 3D Point Cloud Shapes from Part Tree Conditions

Shape generation via learning an adaptive multimodal prior

Learning Part Generation and Assembly for Structure-aware Shape Synthesis

ShapeCaptioner: Generative Caption Network for 3D Shapes by Learning a Mapping from Parts Detected in Multiple Views to Sentences

Michelangelo: Conditional 3D Shape Generation based on Shape-Image-Text Aligned Latent Representation

Conditional Single-View Shape Generation for Multi-View Stereo Reconstruction.

The Shape Part Slot Machine: Contact-based Reasoning for Generating 3D Shapes from Parts

Learning a Probabilistic Latent Space of Object Shapes via 3D Generative-Adversarial Modeling

A Multi-Modal Context Reasoning Approach for Conditional Inference on Joint Textual and Visual Clues

EditVAE: Unsupervised Part-Aware Controllable 3D Point Cloud Shape Generation

A Multi-scale Generative Model for Animate Shapes and Parts.

A Generative Shape Compositional Framework to Synthesize Populations of Virtual Chimeras

3D Semantic Subspace Traverser: Empowering 3D Generative Model with Shape Editing Capability

NeuSDFusion: A Spatial-Aware Generative Model for 3D Shape Completion, Reconstruction, and Generation

Multimodal Conditional 3D Face Geometry Generation

Learning Versatile 3D Shape Generation with Improved AR Models

CC3D: Layout-Conditioned Generation of Compositional 3D Scenes

Learning Versatile 3D Shape Generation with Improved Auto-regressive Models.