Abstract:A practical approach to realizing the comprehensive perception of the surrounding environment is to use a multi-modal fusion method based on various types of vehicular sensors. In clear weather, the camera and LiDAR can provide high-resolution images and point clouds that can be utilized for 3D object detection. However, in foggy weather, the propagation of light is affected by the fog in the air. Consequently, both images and point clouds become distorted to varying degrees. Thus, it is challenging to implement accurate detection in adverse weather conditions. Compared to cameras and LiDAR, Radar possesses strong penetrating power and is not affected by fog. Therefore, this paper proposes a novel two-stage detection framework called “TransFusion”, which leverages LiDAR and Radar fusion to solve the problem of environment perception in foggy weather. The proposed framework is composed of Multi-modal Rotate Region Proposal Network (MM-RRPN) and Multi-modal Refine Network (MM-RFN). Specifically, Spatial Vision Transformer (SVT) and Cross-Modal Attention Mechanism (CMAM) are introduced in the MM-RRPN to improve the robustness of the algorithm in foggy weather. Furthermore, Temporal-Spatial Memory Fusion (TSMF) module in MM-RFN is employed to fuse the spatial-temporal prior information. In addition, the Multi-branches Combination Loss function (MC-Loss) is designed to efficiently supervise the learning of the network. Extensive experiments were conducted on Oxford Radar RobotCar (ORR) dataset. The experimental results show that the proposed algorithm has excellent performance in both foggy and clear weather. Especially in foggy weather, the proposed TransFusion achieves 85.31mAP, outperforming all other competing approaches. The demo is available at: https://youtu.be/ ugjIYHLgn98.

TransRadar: Adaptive-Directional Transformer for Real-Time Multi-View Radar Semantic Segmentation

RaViDeep: Target Detection Based on Deep Fusion of Radar and Vision in Berthing Scenarios

RCTrans: Radar-Camera Transformer via Radar Densifier and Sequential Decoder for 3D Object Detection

Radar Instance Transformer: Reliable Moving Instance Segmentation in Sparse Radar Point Clouds

RadarFormer: Lightweight and Accurate Real-Time Radar Object Detection Model

Radar and Camera Fusion for Multi-Task Sensing in Autonomous Driving

Mask-RadarNet: Enhancing Transformer With Spatial-Temporal Semantic Context for Radar Object Detection in Autonomous Driving

Semantic Segmentation of Radar Detections using Convolutions on Point Clouds

Deep Instance Segmentation with Automotive Radar Detection Points

TransCAR: Transformer-based Camera-And-Radar Fusion for 3D Object Detection

CenterTransFuser: radar point cloud and visual information fusion for 3D object detection

Spatial and Temporal Awareness Network for Semantic Segmentation on Automotive Radar Point Cloud

TransFusion: Multi-Modal Robust Fusion for 3D Object Detection in Foggy Weather Based on Spatial Vision Transformer

A Neural Network Based System for Efficient Semantic Segmentation of Radar Point Clouds

Simultaneous Clutter Detection and Semantic Segmentation of Moving Objects for Automotive Radar Data

Deep Instance Segmentation with High-Resolution Automotive Radar.

Radar Enlighten the Dark: Enhancing Low-Visibility Perception for Automated Vehicles with Camera-Radar Fusion

RADIANT: Radar-Image Association Network for 3D Object Detection

RadarNet: Exploiting Radar for Robust Perception of Dynamic Objects

ROFusion: Efficient Object Detection using Hybrid Point-wise Radar-Optical Fusion

Deep Instance Segmentation with High-Resolution Automotive Radar