Abstract:The fusion of camera and LiDAR perception has become a research focal point in the autonomous driving field. Existing image–point cloud fusion algorithms are overly complex, and processing large amounts of 3D LiDAR point cloud data requires high computational power, which poses challenges for practical applications. To overcome the above problems, herein, we propose an Instance Segmentation Frustum (ISF)–PointPillars method. Within the framework of our method, input data are derived from both a camera and LiDAR. RGB images are processed using an enhanced 2D object detection network based on YOLOv8, thereby yielding rectangular bounding boxes and edge contours of the objects present within the scenes. Subsequently, the rectangular boxes are extended into 3D space as frustums, and the 3D points located outside them are removed. Afterward, the 2D edge contours are also extended to frustums to filter the remaining points from the preceding stage. Finally, the retained points are sent to our improved 3D object detection network based on PointPillars, and this network infers crucial information, such as object category, scale, and spatial position. In pursuit of a lightweight model, we incorporate attention modules into the 2D detector, thereby refining the focus on essential features, minimizing redundant computations, and enhancing model accuracy and efficiency. Moreover, the point filtering algorithm substantially diminishes the volume of point cloud data while concurrently reducing their dimensionality, thereby ultimately achieving lightweight 3D data. Through comparative experiments on the KITTI dataset, our method outperforms traditional approaches, achieving an average precision (AP) of 88.94% and bird’s-eye view (BEV) accuracy of 90.89% in car detection.

Painted PIXOR: A Real-Time Multi-Sensor Fusion Object Detection Pipeline for Autonomous Vehicles

A Fusion Method Aiming at Environmental Perception of Autonomous Vehicle Based on Visual Scheme

Point Cloud Painting for 3D Object Detection with Camera and Automotive 3+1D RADAR Fusion

Real time object detection using LiDAR and camera fusion for autonomous driving

FusionPainting: Multimodal Fusion with Adaptive Attention for 3D Object Detection

A Multi-view 3D Vehicle Detection Method Based On Novel 3D Proposal Generation Method

SemanticVoxels: Sequential Fusion for 3D Pedestrian Detection using LiDAR Point Cloud and Semantic Segmentation

RI-Fusion: 3D Object Detection Using Enhanced Point Features With Range-Image Fusion for Autonomous Driving.

PMPF: Point-Cloud Multiple-Pixel Fusion-Based 3D Object Detection for Autonomous Driving

Real-Time Vehicle Detection Framework Based on the Fusion of LiDAR and Camera

PAI3D: Painting Adaptive Instance-Prior for 3D Object Detection

3D Vehicle Detection Using Multi-Level Fusion From Point Clouds and Images

Object Detection Using Multi-Sensor Fusion Based on Deep Learning

PI-RCNN: An Efficient Multi-sensor 3D Object Detector with Point-based Attentive Cont-conv Fusion Module

ROFusion: Efficient Object Detection using Hybrid Point-wise Radar-Optical Fusion

Enhancing 3D object detection through multi-modal fusion for cooperative perception

Accelerating Point-Voxel Representation of 3-D Object Detection for Automatic Driving

PV-SSD: A Multi-Modal Point Cloud Feature Fusion Method for Projection Features and Variable Receptive Field Voxel Features

Real-Time Vehicle Object Detection Method Based on Multi-Scale Feature Fusion

Instance Segmentation Frustum–PointPillars: A Lightweight Fusion Algorithm for Camera–LiDAR Perception in Autonomous Driving

DyFusion: Cross-Attention 3D Object Detection with Dynamic Fusion