Abstract:Due to its cost-effectiveness and widespread availability, monocular 3D object detection, which relies solely on a single camera during inference, holds significant importance across various applications, including autonomous driving and robotics. Nevertheless, directly predicting the coordinates of objects in 3D space from monocular images poses challenges. Therefore, an effective solution involves transforming monocular images into LiDAR-like representations and employing a LiDAR-based 3D object detector to predict the 3D coordinates of objects. The key step in this method is accurately converting the monocular image into a reliable point cloud form. In this paper, we present VFMM3D, an innovative framework that leverages the capabilities of Vision Foundation Models (VFMs) to accurately transform single-view images into LiDAR point cloud representations. VFMM3D utilizes the Segment Anything Model (SAM) and Depth Anything Model (DAM) to generate high-quality pseudo-LiDAR data enriched with rich foreground information. Specifically, the Depth Anything Model (DAM) is employed to generate dense depth maps. Subsequently, the Segment Anything Model (SAM) is utilized to differentiate foreground and background regions by predicting instance masks. These predicted instance masks and depth maps are then combined and projected into 3D space to generate pseudo-LiDAR points. Finally, any object detectors based on point clouds can be utilized to predict the 3D coordinates of objects. Comprehensive experiments are conducted on two challenging 3D object detection datasets, KITTI and Waymo. Our VFMM3D establishes a new state-of-the-art performance on both datasets. Additionally, experimental results demonstrate the generality of VFMM3D, showcasing its seamless integration into various LiDAR-based 3D object detectors.

Varifocal Multiview Images: Capturing and Visual Tasks

Extending Depth of Field for Varifocal Multiview Images

Recurrent Volume-Based 3-D Feature Fusion for Real-Time Multiview Object Pose Estimation.

End-to-end Varifocal Multiview Images Coding Framework from Data Acquisition End to Vision Application End

Multi-image-distance Imaging System for Extending Depth-of-field

Fusion of infrared and visual images through multiscale hybrid unidirectional total variation

Multiview and light-field reconstruction algorithms for 360° multiple-projector-type 3D display.

Multifocal multiview imaging and data compression based on angular–focal–spatial representation

Vision field capture for advanced 3DTV applications

Hybrid-MVS: Robust Multi-View Reconstruction with Hybrid Optimization of Visual and Depth Cues

Vision field capturing and its applications in 3DTV

Non-reference Virtual View Quality Evaluation of MVD

Fast Automatic Elimination of Vertical Parallax of Multiview Images

Joint Structured Sparsity Regularized Multiview Dimension Reduction for Video-Based Facial Expression Recognition.

Voxelized 3D Feature Aggregation for Multiview Detection

Scale-Consistent Fusion: From Heterogeneous Local Sampling to Global Immersive Rendering

Event-based Visible and Infrared Fusion Via Multi-task Collaboration

VFMM3D: Releasing the Potential of Image by Vision Foundation Model for Monocular 3D Object Detection

A Multi-View Fusion Method Via Tensor Learning And Gradient Descent For Image Features

A Wide Field-of-View Light-Field Camera with Adjustable Multiplicity for Practical Applications

VPFusion: Joint 3D Volume and Pixel-Aligned Feature Fusion for Single and Multi-view 3D Reconstruction