Abstract:While 3D MLLMs have achieved significant progress, they are restricted to object and scene understanding and struggle to understand 3D spatial structures at the part level. In this paper, we introduce Kestrel, representing a novel approach that empowers 3D MLLMs with part-aware understanding, enabling better interpretation and segmentation grounding of 3D objects at the part level. Despite its significance, the current landscape lacks tasks and datasets that endow and assess this capability. Therefore, we propose two novel tasks: (1) Part-Aware Point Grounding, the model is tasked with directly predicting a part-level segmentation mask based on user instructions, and (2) Part-Aware Point Grounded Captioning, the model provides a detailed caption that includes part-level descriptions and their corresponding masks. To support learning and evaluating for these tasks, we introduce 3DCoMPaT Grounded Instructions Dataset (3DCoMPaT-GRIN). 3DCoMPaT-GRIN Vanilla, comprising 789k part-aware point cloud-instruction-segmentation mask triplets, is used to evaluate MLLMs' ability of part-aware segmentation grounding. 3DCoMPaT-GRIN Grounded Caption, containing 107k part-aware point cloud-instruction-grounded caption triplets, assesses both MLLMs' part-aware language comprehension and segmentation grounding capabilities. Our introduced tasks, dataset, and Kestrel represent a preliminary effort to bridge the gap between human cognition and 3D MLLMs, i.e., the ability to perceive and engage with the environment at both global and part levels. Extensive experiments on the 3DCoMPaT-GRIN show that Kestrel can generate user-specified segmentation masks, a capability not present in any existing 3D MLLM. Kestrel thus established a benchmark for evaluating the part-aware language comprehension and segmentation grounding of 3D objects. Project page at <a class="link-external link-https" href="https://feielysia.github.io/Kestrel.github.io/" rel="external noopener nofollow">this https URL</a>

3DCoMPaT$^{++}$: An improved Large-scale 3D Vision Dataset for Compositional Recognition

PartNet: A Large-scale Benchmark for Fine-grained and Hierarchical Part-level 3D Object Understanding

3D Part Segmentation via Geometric Aggregation of 2D Visual Features

VEnvision3D: A Synthetic Perception Dataset for 3D Multi-Task Model Research

Multi-Modal Dataset Acquisition for Photometrically Challenging Object

Improving Annotation for 3D Pose Dataset of Fine-Grained Object Categories

TOD3Cap: Towards 3D Dense Captioning in Outdoor Scenes

Kestrel: Point Grounding Multimodal LLM for Part-Aware 3D Vision-Language Understanding

MARVEL-40M+: Multi-Level Visual Elaboration for High-Fidelity Text-to-3D Content Creation

CATER: A diagnostic dataset for Compositional Actions and TEmporal Reasoning

3x2: 3D Object Part Segmentation by 2D Semantic Correspondences

COAT: Measuring Object Compositionality in Emergent Representations.

COLA: A Benchmark for Compositional Text-to-image Retrieval

Fine-Grained 3D Shape Classification With Hierarchical Part-View Attention

MMScan: A Multi-Modal 3D Scene Dataset with Hierarchical Grounded Language Annotations

Omni6DPose: A Benchmark and Model for Universal 6D Object Pose Estimation and Tracking

DataComp: In search of the next generation of multimodal datasets

GAPartManip: A Large-scale Part-centric Dataset for Material-Agnostic Articulated Object Manipulation

A 6DoF Pose Estimation Dataset and Network for Multiple Parametric Shapes in Stacked Scenarios

Grounded Compositional and Diverse Text-to-3D with Pretrained Multi-View Diffusion Model

Omni6D: Large-Vocabulary 3D Object Dataset for Category-Level 6D Object Pose Estimation