Abstract:Recent works on open-vocabulary 3D instance segmentation show strong promise, but at the cost of slow inference speed and high computation requirements. This high computation cost is typically due to their heavy reliance on 3D clip features, which require computationally expensive 2D foundation models like Segment Anything (SAM) and CLIP for multi-view aggregation into 3D. As a consequence, this hampers their applicability in many real-world applications that require both fast and accurate predictions. To this end, we propose a fast yet accurate open-vocabulary 3D instance segmentation approach, named Open-YOLO 3D, that effectively leverages only 2D object detection from multi-view RGB images for open-vocabulary 3D instance segmentation. We address this task by generating class-agnostic 3D masks for objects in the scene and associating them with text prompts. We observe that the projection of class-agnostic 3D point cloud instances already holds instance information; thus, using SAM might only result in redundancy that unnecessarily increases the inference time. We empirically find that a better performance of matching text prompts to 3D masks can be achieved in a faster fashion with a 2D object detector. We validate our Open-YOLO 3D on two benchmarks, ScanNet200 and Replica, under two scenarios: (i) with ground truth masks, where labels are required for given object proposals, and (ii) with class-agnostic 3D proposals generated from a 3D proposal network. Our Open-YOLO 3D achieves state-of-the-art performance on both datasets while obtaining up to $\sim$16$\times$ speedup compared to the best existing method in literature. On ScanNet200 val. set, our Open-YOLO 3D achieves mean average precision (mAP) of 24.7\% while operating at 22 seconds per scene. Code and model are available at <a class="link-external link-http" href="http://github.com/aminebdj/OpenYOLO3D" rel="external noopener nofollow">this http URL</a>.

Algorithmic Performance-Accuracy Trade-off in 3D Vision Applications Using HyperMapper

GPU Accelerated Real-Time Traversability Mapping.

Unifying Terrain Awareness Through Real-Time Semantic Segmentation

Practical Design Space Exploration

Augmented Efficiency: Reducing Memory Footprint and Accelerating Inference for 3D Semantic Segmentation through Hybrid Vision

Memorize What Matters: Emergent Scene Decomposition from Multitraverse

Real-time 3D semantic occupancy prediction for autonomous vehicles using memory-efficient sparse convolution

3D Adapted Random Forest Vision (3DARFV) for Untangling Heterogeneous-Fabric Exceeding Deep Learning Semantic Segmentation Efficiency at the Utmost Accuracy

BEVDet: High-performance Multi-camera 3D Object Detection in Bird-Eye-View

Mind Mappings: Enabling Efficient Algorithm-Accelerator Mapping Space Search

OMU: A Probabilistic 3D Occupancy Mapping Accelerator for Real-time OctoMap at the Edge

Open-YOLO 3D: Towards Fast and Accurate Open-Vocabulary 3D Instance Segmentation

The Bare Necessities: Designing Simple, Effective Open-Vocabulary Scene Graphs

End-to-End Latency Optimization of Multi-view 3D Reconstruction for Disaster Response

MAP-ADAPT: Real-Time Quality-Adaptive Semantic 3D Maps

Toward Mobile 3D Vision.

Predicting Maps Using In-Vehicle Cameras for Data-Driven Intelligent Transport

Balancing Performance and Efficiency in Zero-shot Robotic Navigation

Stereo Matching Accelerator With Re-Computation Scheme and Data-Reused Pipeline for Autonomous Vehicles

Chameleon: Latency and Resolution Aware Task Offloading for Visual-Based Assisted Driving

HybridOcc: NeRF Enhanced Transformer-based Multi-Camera 3D Occupancy Prediction