Abstract:In recent years, vision-centric perception has played a crucial role in autonomous driving tasks, encompassing functions such as 3D detection, map construction, and motion forecasting. However, the deployment of vision-centric approaches in practical scenarios is hindered by substantial latency, often deviating significantly from the outcomes achieved through offline training. This disparity arises from the fact that conventional benchmarks for autonomous driving perception predominantly conduct offline evaluations, thereby largely overlooking the latency concerns prevalent in real-world deployment. Although a few benchmarks have been proposed to address this limitation by introducing effective evaluation methods for online perception, they do not adequately consider the intricacies introduced by the complexity of input information streams. To address this gap, we propose the Autonomous driving Streaming I/O (ASIO) benchmark, aiming to assess the streaming input characteristics and online performance of vision-centric perception in autonomous driving. To facilitate this evaluation across diverse streaming inputs, we initially establish a dataset based on the CARLA Leaderboard. In alignment with real-world deployment considerations, we further develop evaluation metrics based on information complexity specifically tailored for streaming inputs and streaming performance. Experimental results indicate significant variations in model performance and ranking under different major camera deployments, underscoring the necessity of thoroughly accounting for the influences of model latency and streaming input characteristics during real-world deployment. To enhance streaming performance consistently across distinct streaming input features, we introduce a backbone switcher based on the identified streaming input characteristics. Experimental validation demonstrates its efficacy in perpetually improving streaming performance across varying streaming input features.

StreamTrack: real-time meta-detector for streaming perception in full-speed domain driving scenarios

StreamYOLO: Real-time Object Detection for Streaming Perception

Real-time Stereo-based 3D Object Detection for Streaming Perception

Real-time Streaming Perception System for Autonomous Driving

MTD: Multi-Timestep Detector for Delayed Streaming Perception

StreamMOTP: Streaming and Unified Framework for Joint 3D Multi-Object Tracking and Trajectory Prediction

Towards Streaming Perception

Transtreaming: Adaptive Delay-aware Transformer for Real-time Streaming Perception

Multi-modal Streaming 3D Object Detection

Streaming Object Detection for 3-D Point Clouds

Context-Aware Streaming Perception in Dynamic Environments

Benchmarking Perception to Streaming Inputs in Vision-Centric Autonomous Driving

LongShortNet: Exploring Temporal and Semantic Features Fusion in Streaming Perception

Are We Ready for Vision-Centric Driving Streaming Perception? The ASAP Benchmark

Streaming Motion Forecasting for Autonomous Driving

Adaptive Multi-Pedestrian Tracking by Multi-Sensor: Track-to-Track Fusion Using Monocular 3D Detection and MMW Radar

StreamMOS: Streaming Moving Object Segmentation with Multi-View Perception and Dual-Span Memory

Robust 3D Multi-Object Tracking in Adverse Weather with Hard Sample Mining

Multi-Vehicle Tracking Via Real-Time Detection Probes and a Markov Decision Process Policy.

Multi-Stream Siamese and Faster Region-Based Neural Network for Real-Time Object Tracking

StreamMapNet: Streaming Mapping Network for Vectorized Online HD Map Construction