Abstract:Autonomous driving vehicles rely on sensors for the robust perception of their surroundings. Such vehicles are equipped with multiple perceptive sensors with a high level of redundancy to ensure safety and reliability in any driving condition. However, multi-sensor, such as camera, LiDAR, and radar systems raise requirements related to sensor calibration and synchronization, which are the fundamental blocks of any autonomous system. On the other hand, sensor fusion and integration have become important aspects of autonomous driving research and directly determine the efficiency and accuracy of advanced functions such as object detection and path planning. Classical model-based estimation and data-driven models are two mainstream approaches to achieving such integration. Most recent research is shifting to the latter, showing high robustness in real-world applications but requiring large quantities of data to be collected, synchronized, and properly categorized. However, there are two major research gaps in existing works: (i) they lack fusion (and synchronization) of multi-sensors, camera, LiDAR and radar; and (ii) generic scalable, and user-friendly end-to-end implementation. To generalize the implementation of the multi-sensor perceptive system, we introduce an end-to-end generic sensor dataset collection framework that includes both hardware deploying solutions and sensor fusion algorithms. The framework prototype integrates a diverse set of sensors, such as camera, LiDAR, and radar. Furthermore, we present a universal toolbox to calibrate and synchronize three types of sensors based on their characteristics. The framework also includes the fusion algorithms, which utilize the merits of three sensors, namely, camera, LiDAR, and radar, and fuse their sensory information in a manner that is helpful for object detection and tracking research. The generality of this framework makes it applicable in any robotic or autonomous applications and suitable for quick and large-scale practical deployment.

Multisensor-Pipeline: A Lightweight, Flexible, and Extensible Framework for Building Multimodal-Multisensor Interfaces

A Multimodal Sensing CMOS Imager Based on Dual‐Focus Imaging(Adv. Sci. 14/2023)

Jointly Optimizing Sensing Pipelines for Multimodal Mixed Reality Interaction

End-to-End Multimodal Sensor Dataset Collection Framework for Autonomous Vehicles

A multimodal approach for dynamic event capture of vehicles and pedestrians

An Affordable Bio-Sensing and Activity Tagging Platform for HCI Research

Multi-Sensor Data Fusion for Real-Time Multi-Object Tracking

Developing a comprehensive framework for multimodal feature extraction

Real-Time Hybrid Multi-Sensor Fusion Framework for Perception in Autonomous Vehicles

Bonseyes AI Pipeline -- bringing AI to you. End-to-end integration of data, algorithms and deployment tools

ISSP-Net: An Interactive Spatial-Spectral Perception Network for Multimodal Classification

3D high-resolution multimodal imaging system for real-time applications

A Wearable Multi-Modal Bio-Sensing System Towards Real-World Applications

Graph-Based Multi-Modal Sensor Fusion for Autonomous Driving

Mplatform: A Reconfigurable Architecture and Efficient Data Sharing Mechanism for Modular Sensor Nodes

Multiscale Cloud-based Pipeline for Neuronal Electrophysiology Analysis and Visualization

Single Frame Semantic Segmentation Using Multi-Modal Spherical Images

A Modular Platform For Collaborative, Distributed Sensor Fusion

A multisensory Interaction Framework for Human-Cyber–Physical System based on Graph Convolutional Networks

Automated pipeline for denoising, missing data processing, and feature extraction for signals acquired via wearable devices in multiple sclerosis and amyotrophic lateral sclerosis applications