Detection of Dangerous Human Behavior by Using Optical Flow and Hybrid Deep Learning

Laith Mohammed Salim,Yuksel Celik

DOI: https://doi.org/10.3390/electronics13112116

IF: 2.9

2024-05-30

Electronics

Abstract:Dangerous human behavior in the driving sense may cause traffic accidents and even cause economic losses and casualties. Accurate identification of dangerous human behavior can prevent potential risks. To solve the problem of difficulty retaining the temporal characteristics of the existing data, this paper proposes a human behavior recognition model based on utilized optical flow and hybrid deep learning model-based 3D CNN-LSTM in stacked autoencoder and uses the abnormal behavior of humans in real traffic scenes to verify the proposed model. This model was tested using HMDB51 datasets and JAAD dataset and compared with the recent related works. For a quantitative test, the HMDB51 dataset was used to train and test models for human behavior. Experimental results show that the proposed model achieved good accuracy of about 86.86%, which outperforms recent works. For qualitative analysis, we depend on the initial annotations of walking movements in the JAAD dataset to streamline the annotating process to identify transitions, where we take into consideration flow direction, if it is cross-vehicle motion (to be dangerous) or if it is parallel to vehicle motion (to be of no danger). The results show that the model can effectively identify dangerous behaviors of humans and then test on the moving vehicle scene.

engineering, electrical & electronic,computer science, information systems,physics, applied

What problem does this paper attempt to address?

### Problems the Paper Aims to Solve

The main goal of this paper is to detect dangerous human behaviors in driving scenarios by using optical flow technology and a hybrid deep learning model (based on a 3D CNN-LSTM stacked autoencoder).

#### Specific Problems
1. **Identifying Dangerous Behaviors**: Accurately identifying dangerous human behaviors in driving scenarios, such as pedestrian crossing behaviors, to prevent potential risks.
2. **Preserving Temporal Features**: Addressing the issue of existing data struggling to preserve temporal features by proposing a method that combines optical flow technology and a hybrid deep learning model to improve detection accuracy.
3. **Empirical Validation**: Testing the model using the HMDB51 and JAAD datasets and comparing it with recent related works to demonstrate its superiority.

#### Method Overview
1. **Model Architecture**: Proposing a stacked autoencoder model based on 3D CNN-LSTM to extract temporal and spatial features from video frames.
2. **Optical Flow Technology**: Utilizing optical flow technology to calculate motion vectors between adjacent frames, further enhancing the feature extraction effect.
3. **Experimental Validation**: Conducting experiments on multiple datasets, including HMDB51 and JAAD, to validate the model's effectiveness and accuracy.

#### Main Contributions
1. **New Model**: Proposing a new human behavior recognition system based on a hybrid 3D bidirectional CNN-LSTM stacked autoencoder.
2. **Feature Extraction**: Choosing a stacked autoencoder neural network for feature extraction, improving the effectiveness of feature selection.
3. **Empirical Results**: Achieving an accuracy of approximately 86.86% on the HMDB51 dataset, outperforming recent related studies.

Through these methods and technologies, this paper aims to solve the problem of identifying dangerous human behaviors in driving scenarios and proposes an efficient and accurate solution.