Abstract:Understanding driver activity is vital for in-vehicle systems that aim to reduce the incidence of car accidents rooted in cognitive distraction. Automating real-time behavior recognition while ensuring actions classification with high accuracy is however challenging, given the multitude of circumstances surrounding drivers, the unique traits of individuals, and the computational constraints imposed by in-vehicle embedded platforms. Prior work fails to jointly meet these runtime/accuracy requirements and mostly rely on a single sensing modality, which in turn can be a single point of failure. In this paper, we harness the exceptional feature extraction abilities of deep learning and propose a dedicated Interwoven Deep Convolutional Neural Network (InterCNN) architecture to tackle the problem of accurate classification of driver behaviors in real-time. The proposed solution exploits information from multi-stream inputs, i.e., in-vehicle cameras with different fields of view and optical flows computed based on recorded images, and merges through multiple fusion layers abstract features that it extracts. This builds a tight ensembling system, which significantly improves the robustness of the model. In addition, we introduce a temporal voting scheme based on historical inference instances, in order to enhance the classification accuracy. Experiments conducted with a dataset that we collect in a mock-up car environment demonstrate that the proposed InterCNN with MobileNet convolutional blocks can classify 9 different behaviors with 73.97% accuracy, and 5 'aggregated' behaviors with 81.66% accuracy. We further show that our architecture is highly computationally efficient, as it performs inferences within 15 ms, which satisfies the real-time constraints of intelligent cars. Nevertheless, our InterCNN is robust to lossy input, as the classification remains accurat- when two input streams are occluded.

Driving Behaviour Recognition from Still Images by Using Multi-Stream Fusion CNN.

Vehicle Behavior Recognition using Multi-Stream 3D Convolutional Neural Network

Driving Behaviour Style Study with a Hybrid Deep Learning Framework Based on GPS Data

Driver Behavior Recognition via Interwoven Deep Convolutional Neural Nets With Multi-Stream Inputs

Driver behavior recognition based on deep convolutional neural networks

Application of Deep Learning Methods in Driver Behavior Recognition

Driving Behavior Recognition Algorithm Combining Attention Mechanism and Lightweight Network

Driving Behaviour Characterisation by Using Phase‐space Reconstruction and Pre‐trained Convolutional Neural Network

A Driving Behavior Recognition Model with Bi-LSTM and Multi-Scale CNN

Driver Activity Recognition for Intelligent Vehicles: A Deep Learning Approach

Multi-CNN and Decision Tree Based Driving Behavior Evaluation

An abnormal driving behavior recognition algorithm based on the temporal convolutional network and soft thresholding

A Hybrid Approach for Driving Behavior Recognition: Integration of CNN and Transformer-Encoder with EEG Data

End-to-End Driving Activities and Secondary Tasks Recognition Using Deep Convolutional Neural Network and Transfer Learning

Real-Time Driving Behavior Identification Based on Multi-Source Data Fusion

RFDANet: an FMCW and TOF radar fusion approach for driver activity recognition using multi-level attention based CNN and LSTM network

Recognition method of abnormal driving behavior using the bidirectional gated recurrent unit and convolutional neural network

Driving Behavior Tracking and Recognition Based on Multisensors Data Fusion

Evolution and clinical importance of fibrosis in HCM.

A Deep Learning Framework for Driving Behavior Identification on In-Vehicle CAN-BUS Sensor Data

Driver behavior detection and classification using deep convolutional neural networks