Abstract:In the automotive industry, intelligent monitoring systems for advanced human-vehicle interaction aimed at enhancing the safety of drivers and passengers represent a rapidly growing area of research. Safe driving behavior relies on the driver's awareness of the road context, enabling them to make appropriate decisions and act consistently in anomalous circumstances. A potentially dangerous situation can arise when an emergency vehicle rapidly approaches with sirens blaring. In such cases, it is crucial for the driver to perform the correct maneuvers to prioritize the emergency vehicle. For this purpose, an Advanced Driver Assistance System (ADAS) can provide timely alerts to the driver about an approaching emergency vehicle. In this work, we present a driver-assistance prototype that leverages multimodal information from an integrated audio and video monitoring system. In the initial stage, sound analysis technologies based on computational audio processing are employed to recognize the proximity of an emergency vehicle based on the sound of its siren. When such an event occurs, an in-vehicle monitoring system is activated, analyzing the driver's facial patterns using deep-learning-based algorithms to assess their awareness. This work illustrates the design of such a prototype, presenting the hardware technologies, the software architecture, and the deep-learning algorithms for audio and video data analysis that make the driver-assistance prototype operational in a commercial car. At this initial experimental stage, the algorithms for analyzing the audio and video data have yielded promising results. The area under the precision-recall curve for siren identification stands at 0.92, while the accuracy in evaluating driver gaze orientation reaches 0.97. In conclusion, engaging in research within this field has the potential to significantly improve road safety by increasing driver awareness and facilitating timely and well-informed reactions to crucial situations. This could substantially reduce risks and ultimately protect lives on the road.

A multimodal approach for dynamic event capture of vehicles and pedestrians

Online Multipedestrian Tracking Based on Fused Detections of Millimeter Wave Radar and Vision

A Multiple Vehicle Sensing Approach for Collision Avoidance in Progressively Deployed Vehicle Networks.

Adaptive Multi-Pedestrian Tracking by Multi-Sensor: Track-to-Track Fusion Using Monocular 3D Detection and MMW Radar

Spatio-Contextual Deep Network Based Multimodal Pedestrian Detection For Autonomous Driving

MCMSys: Multimodal Data Closed-Loop Management System for Autonomous Driving

Study on Multi-Heterogeneous Sensor Data Fusion Method Based on Millimeter-Wave Radar and Camera

Multimodal Collaboration Networks for Geospatial Vehicle Detection in Dense, Occluded, and Large-Scale Events

Revisiting Modality Imbalance In Multimodal Pedestrian Detection

A multimodal vision sensor for autonomous driving

Multi-modal Sensor Registration for Vehicle Perception via Deep Neural Networks

Can multimodal sensing detect and localize transient events?

Dynamic Multi-LiDAR Based Multiple Object Detection and Tracking

Multimodal feature fusion for robust event detection in web videos

End-to-End Multimodal Sensor Dataset Collection Framework for Autonomous Vehicles

MIPD: A Multi-sensory Interactive Perception Dataset for Embodied Intelligent Driving

Multimodal Classification of Urban Micro-Events

An advanced multimodal driver-assistance prototype for emergency-vehicle detection

The Multimodal Driver Monitoring Database: A Naturalistic Corpus to Study Driver Attention

An Empirical Evaluation of Neural and Neuro-symbolic Approaches to Real-time Multimodal Complex Event Detection