Abstract:Accurate obstacle identification represents a fundamental challenge within the scope of near-field perception for autonomous driving. Conventionally, fisheye cameras are frequently employed for comprehensive surround-view perception, including rear-view obstacle localization. However, the performance of such cameras can significantly deteriorate in low-light conditions, during nighttime, or when subjected to intense sun glare. Conversely, cost-effective sensors like ultrasonic sensors remain largely unaffected under these conditions. Therefore, we present, to our knowledge, the first end-to-end multimodal fusion model tailored for efficient obstacle perception in a bird's-eye-view (BEV) perspective, utilizing fisheye cameras and ultrasonic sensors. Initially, ResNeXt-50 is employed as a set of unimodal encoders to extract features specific to each modality. Subsequently, the feature space associated with the visible spectrum undergoes transformation into BEV. The fusion of these two modalities is facilitated via concatenation. At the same time, the ultrasonic spectrum-based unimodal feature maps pass through content-aware dilated convolution, applied to mitigate the sensor misalignment between two sensors in the fused feature space. Finally, the fused features are utilized by a two-stage semantic occupancy decoder to generate grid-wise predictions for precise obstacle perception. We conduct a systematic investigation to determine the optimal strategy for multimodal fusion of both sensors. We provide insights into our dataset creation procedures, annotation guidelines, and perform a thorough data analysis to ensure adequate coverage of all scenarios. When applied to our dataset, the experimental results underscore the robustness and effectiveness of our proposed multimodal fusion approach.

Surround-View Fisheye Camera Perception for Automated Driving: Overview, Survey & Challenges

Surround-view Fisheye Camera Perception for Automated Driving: Overview, Survey and Challenges

Surround-View Fisheye Optics in Computer Vision and Simulation: Survey and Challenges

3-D Surround View for Advanced Driver Assistance Systems.

Surround-View Fisheye BEV-Perception for Valet Parking: Dataset, Baseline and Distortion-Insensitive Multi-Task Framework

FisheyeDetNet: 360° Surround view Fisheye Camera based Object Detection System for Autonomous Driving

SVDistNet: Self-Supervised Near-Field Distance Estimation on Surround View Fisheye Cameras

3D visual perception for self-driving cars using a multi-camera system: Calibration, mapping, localization, and obstacle detection

Capture Of Vehicle Surroundings Using A Pair Of Panoramic Annular Lens Cameras

Object Detection and Localization in 3D Environment by Fusing Raw Fisheye Image and Attitude Data

Disentangling and Vectorization: A 3D Visual Perception Approach for Autonomous Driving Based on Surround-View Fisheye Cameras

Surround-View Vision-based 3D Detection for Autonomous Driving: A Survey

SynWoodScape: Synthetic Surround-view Fisheye Camera Dataset for Autonomous Driving

Restricted Deformable Convolution based Road Scene Semantic Segmentation Using Surround View Cameras

Automatic Surround Camera Calibration Method in Road Scene for Self-driving Car

DaF-BEVSeg: Distortion-aware Fisheye Camera based Bird's Eye View Segmentation with Occlusion Reasoning

Fisheye Camera and Ultrasonic Sensor Fusion For Near-Field Obstacle Perception in Bird's-Eye-View

Delving Into the Devils of Bird's-Eye-View Perception: A Review, Evaluation and Recipe

Enhanced Parking Perception by Multi-Task Fisheye Cross-view Transformers

A Detection and Tracking System for Fisheye Videos from Traffic Intersections