Multimodal Driver State Modeling through Unsupervised Learning

Arash Tavakoli,Arsalan Heydarian

DOI: https://doi.org/10.48550/arXiv.2110.01727

2021-10-05

Abstract:Naturalistic driving data (NDD) can help understand drivers' reactions to each driving scenario and provide personalized context to driving behavior. However, NDD requires a high amount of manual labor to label certain driver's state and behavioral patterns. Unsupervised analysis of NDD can be used to automatically detect different patterns from the driver and vehicle data. In this paper, we propose a methodology to understand changes in driver's physiological responses within different driving patterns. Our methodology first decomposes a driving scenario by using a Bayesian Change Point detection model. We then apply the Latent Dirichlet Allocation method on both driver state and behavior data to detect patterns. We present two case studies in which vehicles were equipped to collect exterior, interior, and driver behavioral data. Four patterns of driving behaviors (i.e., harsh brake, normal brake, curved driving, and highway driving), as well as two patterns of driver's heart rate (HR) (i.e., normal vs. abnormal high HR), and gaze entropy (i.e., low versus high), were detected in these two case studies. The findings of these case studies indicated that among our participants, the drivers' HR had a higher fraction of abnormal patterns during harsh brakes, accelerating and curved driving. Additionally, free-flow driving with close to zero accelerations on the highway was accompanied by more fraction of normal HR as well as a lower gaze entropy pattern. With the proposed methodology we can better understand variations in driver's psychophysiological states within different driving scenarios. The findings of this work, has the potential to guide future autonomous vehicles to take actions that are fit to each specific driver.

Human-Computer Interaction

What problem does this paper attempt to address?

The problem that this paper attempts to solve is how to automatically detect the changes in drivers' psychophysiological responses in different driving modes from naturalistic driving data (NDD) through unsupervised learning methods. Specifically, the research aims to:

1. **Reduce manual annotation work**: Naturalistic driving data usually requires a large amount of manual labor to annotate specific driver states and behavior patterns. Through unsupervised learning methods, these patterns can be automatically detected, reducing the need for manual annotation.
2. **Understand drivers' reactions in different driving scenarios**: By analyzing physiological indicators such as drivers' heart rates and gaze entropy, understand the changes in drivers' psychophysiological states under different driving behaviors (such as sudden braking, normal braking, corner driving, and highway driving).
3. **Personalize driving behaviors**: By analyzing drivers' physiological and behavioral data, personalized driving contexts can be provided for each driver, thereby guiding future autonomous vehicles to better adapt to the specific needs of each driver.

### Method Overview

1. **Data decomposition**: Use the Bayesian Change Point (BCP) model to decompose the driving scenarios into multiple small segments to detect changes in the distribution of vehicle dynamics data.
2. **Pattern detection**: Apply the Latent Dirichlet Allocation (LDA) method to perform pattern detection on driver state and behavior data.
3. **Case study**: Verify the effectiveness of the method through two case studies. The first case study used high - frequency driving behavior data and driver sensing data (heart rate and gaze), and detected four driving modes (sudden braking, normal braking, free driving on the highway, corner driving) and two driver state modes (abnormally high heart rate vs. baseline heart rate, low gaze entropy vs. high gaze entropy). The second case study was extended to a larger data set, including 12 participants, and the results were similar to those of the first case study.

### Formula Presentation

1. **Bayesian change point detection**:
- Posterior density formula:
\[
f(\rho | X) \propto f(X | \rho) f(\rho)
\]
- Prior concentration density formula:
\[
f(\rho | p) = p^{b - 1}(1 - p)^{n - b}
\]
- Joint density formula:
\[
f(X_{ij})=\left(\frac{1}{2\pi\sigma^2}\right)^{(j - i)/2}\left(\frac{\sigma^2_0+\sigma^2}{\sigma^2}\right)^{1/2}\exp(V_{ij})
\]
where:
\[
V_{ij}=-\frac{\sum_{l = i + 1}^{j}(X_l-\hat{X}_{ij})^2}{2\sigma^2}-\frac{(j - i)(\hat{X}_{ij}-\mu_0)^2}{2(\sigma^2+\sigma^2_0)}
\]
\(\hat{X}_{ij}\) is the mean of the observed values within the interval.

2. **Conditional entropy (GTE) calculation**:
- Conditional entropy formula:
\[
\hat{H}_t=-\sum_{i\in S}\pi_i\sum_{j\in S}p_{ij}\log_2p_{ij}
\]
where \(p_{ij}\) is the probability of transitioning from region of interest \(i\) to \(j\), and \(\pi_i\) is the steady - state probability of \(i\).

Through these methods and formulas, the research can effectively extract drivers' state and behavior patterns from naturalistic driving data, providing support for further personalized driving and autonomous driving technologies.