Abstract:This paper explores Deep Learning (DL) methods that are used or have the potential to be used for traffic video analysis, emphasizing driving safety for both Autonomous Vehicles (AVs) and human-operated vehicles. We present a typical processing pipeline, which can be used to understand and interpret traffic videos by extracting operational safety metrics and providing general hints and guidelines to improve traffic safety. This processing framework includes several steps, including video enhancement, video stabilization, semantic and incident segmentation, object detection and classification, trajectory extraction, speed estimation, event analysis, modeling and anomaly detection. Our main goal is to guide traffic analysts to develop their own custom-built processing frameworks by selecting the best choices for each step and offering new designs for the lacking modules by providing a comparative analysis of the most successful conventional and DL-based algorithms proposed for each step. We also review existing open-source tools and public datasets that can help train DL models. To be more specific, we review exemplary traffic problems and mentioned requires steps for each problem. Besides, we investigate connections to the closely related research areas of drivers' cognition evaluation, Crowd-sourcing-based monitoring systems, Edge Computing in roadside infrastructures, Automated Driving Systems (ADS)-equipped vehicles, and highlight the missing gaps. Finally, we review commercial implementations of traffic monitoring systems, their future outlook, and open problems and remaining challenges for widespread use of such systems.

What problem does this paper attempt to address?

### Problems the Paper Attempts to Solve The paper "Deep Learning for Traffic Safety Management: A Forward-Looking Review" aims to explore deep learning (DL) methods used for traffic video analysis, with a particular emphasis on the safety of autonomous vehicles (AVs) and human-driven vehicles. The main objectives of the paper include: 1. **Propose a Typical Processing Workflow**: Introduce a typical processing workflow that can understand, interpret traffic videos, and extract operational safety metrics, providing general guidance and recommendations to improve traffic safety. 2. **Cover Multiple Processing Steps**: Provide a detailed description of the various steps in the processing framework, including video enhancement, video stabilization, semantic and event segmentation, object detection and classification, trajectory extraction, speed estimation, event analysis, modeling, and anomaly detection. 3. **Guide Traffic Analysts**: Assist traffic analysts in developing their own customized processing frameworks by selecting the best solutions for each step and offering new design ideas for missing modules, along with comparative analysis of algorithms. 4. **Review Open Source Tools and Public Datasets**: Introduce existing open-source tools and public datasets that can help train deep learning models. 5. **Review Related Research Areas**: Explore the connections with closely related research areas such as driver cognitive assessment, crowdsourced monitoring systems, edge computing for roadside infrastructure, and vehicles equipped with autonomous driving systems, and point out the gaps in these areas. 6. **Review Commercial Implementations**: Review the commercial implementations of existing traffic monitoring systems, forecast their future development directions, and discuss the open issues and remaining challenges faced in the widespread use of these systems. In summary, this paper is dedicated to improving traffic safety through the application of deep learning methods in traffic video analysis, providing comprehensive guidance and tool support for traffic analysts and researchers.

Deep Learning Serves Traffic Safety Analysis: A Forward-looking Review

Short-Term Forecasting of Traffic Flow and Speed: A Deep Learning Approach

Vehicle-Related Scene Understanding Using Deep Learning

Deep Learning for Safe Autonomous Driving: Current Challenges and Future Directions

Deep Learning for Intelligent Traffic Sensing and Prediction: Recent Advances and Future Challenges

A system of vision sensor based deep neural networks for complex driving scene analysis in support of crash risk assessment and prevention

Highly Accurate Deep Learning Models for Estimating Traffic Characteristics from Video Data

Artificial Intelligence Enabled Traffic Monitoring System

Real-time Traffic Surveillance and Detection using Deep Learning and Computer Vision Techniques

Deep Learning-Based Pedestrian Detection in Autonomous Vehicles: Substantial Issues and Challenges

Real-time Hazard Detection in Autonomous Vehicles: Deep Learning Approaches for Enhancing Road Safety

AI-Based Vehicular Network toward 6G and IoT: Deep Learning Approaches

An Empirical Evaluation of Deep Learning on Highway Driving

Involvement of Deep Learning for Vision Sensor-Based Autonomous Driving Control: A Review

Scene Detection Classification and Tracking for Self-Driven Vehicle

Exploring the impact of traffic signal control and connected and automated vehicles on intersections safety: A deep reinforcement learning approach

Deep Learning Techniques for Vehicle Detection and Classification from Images/Videos: A Survey

A Survey on Deep Learning-Based Traffic Signal Control

Deep learning adversarial attacks and defenses in autonomous vehicles: a systematic literature review from a safety perspective

Overview of Deep Learning Intelligent Driving Methods

DL-Traff: Survey and Benchmark of Deep Learning Models for Urban Traffic Prediction