Abstract:In this work, we propose a novel one-stage and keypoint-based framework for monocular 3D object detection using only RGB images, called KM3D-Net. 2D detection only requires a deep neural network to predict 2D properties of objects, as it is a semanticity-aware task. For image-based 3D detection, we argue that the combination of a deep neural network and geometric constraints are needed to synergistically estimate appearance-related and spatial-related information. Here, we design a fully convolutional model to predict object keypoints, dimension, and orientation, and combine these with perspective geometry constraints to compute position attributes. Further, we reformulate the geometric constraints as a differentiable version and embed this in the network to reduce running time while maintaining the consistency of model outputs in an end-to-end fashion. Benefiting from this simple structure, we propose an effective semi-supervised training strategy for settings where labeled training data are scarce. In this strategy, we enforce a consensus prediction of two shared-weights KM3D-Net for the same unlabeled image under different input augmentation conditions and network regularization. In particular, we unify the coordinate-dependent augmentations as the affine transformation for the differential recovering position of objects, and propose a keypoint-dropout module for network regularization. Our model only requires RGB images, without synthetic data, instance segmentation, CAD model, or depth generator. Extensive experiments on the popular KITTI 3D detection dataset indicate that the KM3D-Net surpasses state-of-the-art methods by a large margin in both efficiency and accuracy. And also, to the best of our knowledge, this is the first application of semi-supervised learning in monocular 3D object detection. We surpass most of the previous fully supervised methods with only 13% labeled data on KITTI.

Deep Learning for Vanishing Point Detection Using an Inverse Gnomonic Projection

NeurVPS: Neural Vanishing Point Scanning Via Conic Convolution.

Vanishing point detection with convolutional neural networks

Optimal Vanishing Point Estimation with Performance Evaluation

D-VPnet: A Network for Real-time Dominant Vanishing Point Detection in Natural Scenes

MonoDGP: Monocular 3D Object Detection with Decoupled-Query and Geometry-Error Priors

Vanishing Point Detection with Direct and Transposed Fast Hough Transform inside the neural network

Simultaneous Vanishing Point Detection And Camera Calibration From Single Images

Research on the Vanishing Point Detection Method Based on an Improved Lightweight AlexNet Network for Narrow Waterway Scenarios

Three Orthogonal Vanishing Points Estimation In Structured Scenes Using Convolutional Neural Networks

D-NPC: Dynamic Neural Point Clouds for Non-Rigid View Synthesis from Monocular Video

VIEWPOINT ESTIMATION IN IMAGES BY A KEY-POINT BASED DEEP NEURAL NETWORK

A Geometric Approach to Obtain a Bird's Eye View from an Image

Vanishing Point Estimation in Uncalibrated Images with Prior Gravity Direction

Vanishing point estimation inspired by oblique effect in a field environment

DynOMo: Online Point Tracking by Dynamic Online Monocular Gaussian Reconstruction

MonoJSG: Joint Semantic and Geometric Cost Volume for Monocular 3D Object Detection

Monocular 3D Detection With Geometric Constraint Embedding and Semi-Supervised Training

VPDETR: End-to-End Vanishing Point DEtection TRansformers

LeanGaussian: Breaking Pixel or Point Cloud Correspondence in Modeling 3D Gaussians

VET: Visual Error Tomography for Point Cloud Completion and High-Quality Neural Rendering