Abstract:Understanding driver activity is vital for in-vehicle systems that aim to reduce the incidence of car accidents rooted in cognitive distraction. Automating real-time behavior recognition while ensuring actions classification with high accuracy is however challenging, given the multitude of circumstances surrounding drivers, the unique traits of individuals, and the computational constraints imposed by in-vehicle embedded platforms. Prior work fails to jointly meet these runtime/accuracy requirements and mostly rely on a single sensing modality, which in turn can be a single point of failure. In this paper, we harness the exceptional feature extraction abilities of deep learning and propose a dedicated Interwoven Deep Convolutional Neural Network (InterCNN) architecture to tackle the problem of accurate classification of driver behaviors in real-time. The proposed solution exploits information from multi-stream inputs, i.e., in-vehicle cameras with different fields of view and optical flows computed based on recorded images, and merges through multiple fusion layers abstract features that it extracts. This builds a tight ensembling system, which significantly improves the robustness of the model. In addition, we introduce a temporal voting scheme based on historical inference instances, in order to enhance the classification accuracy. Experiments conducted with a dataset that we collect in a mock-up car environment demonstrate that the proposed InterCNN with MobileNet convolutional blocks can classify 9 different behaviors with 73.97% accuracy, and 5 'aggregated' behaviors with 81.66% accuracy. We further show that our architecture is highly computationally efficient, as it performs inferences within 15 ms, which satisfies the real-time constraints of intelligent cars. Nevertheless, our InterCNN is robust to lossy input, as the classification remains accurat- when two input streams are occluded.

Vehicular Environment Identification Based on Channel State Information and Deep Learning

Rainy Environment Identification Based on Channel State Information for Autonomous Vehicles

Vehicle Behavior Recognition using Multi-Stream 3D Convolutional Neural Network

Identification and Recognition of Vehicle Environment Using Artificial Neural Networks

Deep Reinforcement Learning for Channel State Information Prediction in Internet of Vehicles

Channel Estimation Based on Deep Learning in Vehicle-to-Everything Environments

Machine-Learning-Based Scenario Identification Using Channel Characteristics in Intelligent Vehicular Communications

Deep Convolutional Neural Network for Roadway Incident Surveillance Using Audio Data

Learning the Wireless V2I Channels Using Deep Neural Networks

Intelligent Highway Traffic Detection Algorithm based on Deep Learning

Environment-Robust Device-Free Human Activity Recognition With Channel-State-Information Enhancement and One-Shot Learning

Automated Vehicle Recognition with Deep Convolutional Neural Networks

A Variational Learning Approach for Concurrent Distance Estimation and Environmental Identification

Deep Neural Network Based Vehicle Detection and Classification of Aerial Images

Lightweight Deep Learning for Road Environment Recognition

Driver Behavior Recognition via Interwoven Deep Convolutional Neural Nets With Multi-Stream Inputs

Efficient and Deep Vehicle Re-Identification Using Multi-Level Feature Extraction

Robust Environment Perception for Automated Driving: A Unified Learning Pipeline for Visual-Infrared Object Detection

Vehicular Network Intrusion Detection Using a Cascaded Deep Learning Approach with Multi-Variant Metaheuristic

Multitarget Vehicle Tracking and Motion State Estimation Using a Novel Driving Environment Perception System of Intelligent Vehicles

Context-Conditioned Spatio-Temporal Predictive Learning for Reliable V2V Channel Prediction