Abstract:Recent advancements in environmental perception for autonomous vehicles have been driven by deep learning-based approaches. However, effective traffic target detection in complex environments remains a challenging task. This paper presents a novel dual-modal instance segmentation deep neural network (DM-ISDNN) by merging camera and LIDAR data, which can be used to deal with the problem of target detection in complex environments efficiently based on multi-sensor data fusion. Due to the sparseness of the LIDAR point cloud data, we propose a weight assignment function that assigns different weight coefficients to different feature pyramid convolutional layers for the LIDAR sub-network. We compare and analyze the adaptations of early-, middle-, and late-stage fusion architectures in depth. By comprehensively considering the detection accuracy and detection speed, the middle-stage fusion architecture with a weight assignment mechanism, with the best performance, is selected. This work has great significance for exploring the best feature fusion scheme of a multi-modal neural network. In addition, we apply a mask distribution function to improve the quality of the predicted mask. A dual-modal traffic object instance segmentation dataset is established using a 7481 camera and LIDAR data pairs from the KITTI dataset, with 79,118 manually annotated instance masks. To the best of our knowledge, there is no existing instance annotation for the KITTI dataset with such quality and volume. A novel dual-modal dataset, composed of 14,652 camera and LIDAR data pairs, is collected using our own developed autonomous vehicle under different environmental conditions in real driving scenarios, for which a total of 62,579 instance masks are obtained using semi-automatic annotation method. This dataset can be used to validate the detection performance under complex environmental conditions of instance segmentation networks. Experimental results on the dual-modal KITTI Benchmark demonstrate that DM-ISDNN using middle-stage data fusion and the weight assignment mechanism has better detection performance than single- and dual-modal networks with other data fusion strategies, which validates the robustness and effectiveness of the proposed method. Meanwhile, compared to the state-of-the-art instance segmentation networks, our method shows much better detection performance, in terms of AP and F1 score, on the dual-modal dataset collected under complex environmental conditions, which further validates the superiority of our method.

Efficient Automatic Driving Instance Segmentation Method Based on Detection

A Scalable Real-time Semantic Segmentation Network for Autonomous Driving

Multiclass objects detection algorithm using DarkNet-53 and DenseNet for intelligent vehicles

Road Traffic Sign Detection Method Based on RTS R-CNN Instance Segmentation Network

A Joint Object Detection and Semantic Segmentation Model with Cross-Attention and Inner-Attention Mechanisms

OccludedInst: an Efficient Instance Segmentation Network for Automatic Driving Occlusion Scenes

Improved Mask R-CNN Multi-Target Detection and Segmentation for Autonomous Driving in Complex Scenes

Deep Dual-Modal Traffic Objects Instance Segmentation Method Using Camera and LIDAR Data for Autonomous Driving.

TTIS-YOLO: A Traffic Target Instance Segmentation Paradigm for Complex Road Scenarios

CompleteInst: An Efficient Instance Segmentation Network for Missed Detection Scene of Autonomous Driving

Multi-Task Environmental Perception Methods for Autonomous Driving

Research on multitask model of object detection and road segmentation in unstructured road scenes

A Two-Pipeline Instance Segmentation Network via Boundary Enhancement for Scene Understanding

3D Instance Segmentation and Object Detection Framework Based on the Fusion of Lidar Remote Sensing and Optical Image Sensing

Vehicle Detection through Instance Segmentation using Mask R-CNN for Intelligent Vehicle System

YOLOv4-5D: An Effective and Efficient Object Detector for Autonomous Driving

Efficient Vehicle Detection and Optimization in Multi-Graph Mode Considering Multi-Section Tracking Based on Geographic Similarity

L2-LiteSeg: A Real-Time Semantic Segmentation Method for End-to-End Autonomous Driving

Multi-Task Visual Perception for Object Detection and Semantic Segmentation in Intelligent Driving

Attention Mechanism and Detection Box Information Based Real-time Multi-Object Vehicle Detection