Abstract:Holistically understanding an object and its 3D movable parts through visual perception models is essential for enabling an autonomous agent to interact with the world. For autonomous driving, the dynamics and states of vehicle parts such as doors, the trunk, and the bonnet can provide meaningful semantic information and interaction states, which are essential to ensuring the safety of the self-driving vehicle. Existing visual perception models mainly focus on coarse parsing such as object bounding box detection or pose estimation and rarely tackle these situations. In this paper, we address this important autonomous driving problem by solving three critical issues. First, to deal with data scarcity, we propose an effective training data generation process by fitting a 3D car model with dynamic parts to vehicles in real images before reconstructing human-vehicle interaction (VHI) scenarios. Our approach is fully automatic without any human interaction, which can generate a large number of vehicles in uncommon states (VUS) for training deep neural networks (DNNs). Second, to perform fine-grained vehicle perception, we present a multi-task network for VUS parsing and a multi-stream network for VHI parsing. Third, to quantitatively evaluate the effectiveness of our data augmentation approach, we build the first VUS dataset in real traffic scenarios (e.g., getting on/out or placing/removing luggage). Experimental results show that our approach advances other baseline methods in 2D detection and instance segmentation by a big margin (over 8%). In addition, our network yields large improvements in discovering and understanding these uncommon cases. Moreover, we have released the source code, the dataset, and the trained model on Github (https://github.com/zongdai/EditingForDNN).

OBB-based 4D Manipulation of Traffic Sensory Data for Generating Corner Driving Scenarios

A Fusion Method Aiming at Environmental Perception of Autonomous Vehicle Based on Visual Scheme

3-D Surround View for Advanced Driver Assistance Systems.

Simulating Dynamic Driving Behavior in Simulation Test for Unmanned Vehicles via Multi-Sensor Data.

3D Object Visibility Prediction in Autonomous Driving

Autonomous Driving Simulation for Unmanned Vehicles.

DrivingSphere: Building a High-fidelity 4D World for Closed-loop Simulation

OCC-VO: Dense Mapping via 3D Occupancy-Based Visual Odometry for Autonomous Driving

Occlusion-Aware 2D and 3D Centerline Detection for Urban Driving via Automatic Label Generation

Fine-Grained Vehicle Perception via 3D Part-Guided Visual Data Augmentation

DriveDreamer4D: World Models Are Effective Data Machines for 4D Driving Scene Representation

Visual-based On-Road Vehicle Detection: A Transnational Experiment and Comparison

Combined Edge- and Stixel-based Object Detection in 3D Point Cloud

Disentangling and Vectorization: A 3D Visual Perception Approach for Autonomous Driving Based on Surround-View Fisheye Cameras

Scene as Occupancy

Instantaneous Perception of Moving Objects in 3D

3D Instance Segmentation and Object Detection Framework Based on the Fusion of Lidar Remote Sensing and Optical Image Sensing

MapVision: CVPR 2024 Autonomous Grand Challenge Mapless Driving Tech Report

3D Bounding Box Estimation for Autonomous Vehicles by Cascaded Geometric Constraints and Depurated 2D Detections Using 3D Results

LiDAR-based 4D Occupancy Completion and Forecasting