Abstract:Sleep is known to be a key factor in emotional regulation and overall mental health. In this study, we explore the integration of sleep measures from the previous night into wearable-based mood recognition. To this end, we propose NapTune, a novel prompt-tuning framework that utilizes sleep-related measures as additional inputs to a frozen pre-trained wearable time-series encoder by adding and training lightweight prompt parameters to each Transformer layer. Through rigorous empirical evaluation, we demonstrate that the inclusion of sleep data using NapTune not only improves mood recognition performance across different wearable time-series namely ECG, PPG, and EDA, but also makes it more sample-efficient. Our method demonstrates significant improvements over the best baselines and unimodal variants. Furthermore, we analyze the impact of adding sleep-related measures on recognizing different moods as well as the influence of individual sleep-related measures.
What problem does this paper attempt to address?
### What problem does this paper attempt to solve?
This paper aims to solve the problem of how to integrate the sleep data from the previous night into the time - series data of wearable devices to improve the performance of emotion classification (i.e., mood recognition). Specifically, the authors propose a new framework named NapTune, which improves the emotion classification model based on wearable time - series data (such as ECG, PPG, and EDA) by introducing sleep - related indicators from the previous night as additional inputs.
#### Background and Problem Description
1. **Importance of Emotion Recognition**
- Emotion recognition has important applications in fields such as human - computer interaction, healthcare, education, and user experience.
- Wearable devices have become an important tool for emotion recognition because they can continuously and non - intrusively monitor physiological signals (such as electrocardiogram ECG, photoplethysmography PPG, and electrodermal activity EDA).
2. **Impact of Sleep on Emotion**
- Sleep is a key factor affecting emotion regulation and mental health.
- Indicators such as sleep quality and duration are closely related to emotional states, but existing research has not fully incorporated the sleep data from the previous night into automated emotion recognition models.
3. **Data Scarcity**
- The lack of public datasets containing sleep data from the previous night and wearable time - series data of the same user limits the effectiveness of multimodal training.
#### Goals of the NapTune Framework
- **Integrate Sleep Data**: By introducing sleep - related indicators from the previous night (such as time in bed, sleep duration, deep - sleep duration, etc.), enhance the emotion classification model based on wearable time - series data.
- **Efficient Fine - Tuning**: Use lightweight parameter adjustment to pre - train the wearable time - series encoder so that it can handle sleep data without a large amount of paired training data.
- **Performance Improvement**: Experiments prove that adding sleep data can significantly improve the performance of emotion classification, with the F1 score increasing by up to 8%.
### Main Contributions
1. **Propose the NapTune Framework**: An efficient fine - tuning method that enables the frozen wearable time - series encoder to use the sleep data from the previous night as auxiliary input for emotion classification.
2. **Reduce Data Requirements**: Only a small amount of paired data is required to achieve effective multimodal training.
3. **Experimental Verification**: Experiments show that on different types of wearable time - series data (ECG, PPG, EDA), after adding sleep data, the performance of emotion classification is significantly improved, and the F1 score is increased by 9% - 11%.
### Formula Summary
- **Attention Mechanism Formula**
\[
\text{Attention}(Q, K, V)=\text{Softmax}\left(\frac{Q\cdot K^{T}}{\sqrt{d}}\times\text{Mask}\right)V
\]
where \(Q\), \(K\) and \(V\) are the query, key and value matrices respectively, and \(d\) is their dimension.
- **Contrastive Loss Function**
\[
\ell_{p, q}=-\log\frac{\exp(\text{sim}(\mathbf{z}_{p},\mathbf{z}_{q})/\tau)}{\sum_{k = 1}^{2M}1_{k\neq i}\exp(\text{sim}(\mathbf{z}_{p},\mathbf{z}_{k})/\tau)}
\]
where \(\text{sim}\) represents the pairwise dot - product similarity and \(\tau\) is the temperature parameter.
- **Binary Cross - Entropy Loss Function**
\[
\text{BCE}(p,\tilde{p})=-\frac{1}{C}\sum_{j = 1}^{C}[p_{j}\log\tilde{p}_{j}+(1 - p_{j})\log(1-\tilde{p}_{j})]
\]
where \(C\) is the number of classes, and \(p\) and \(\tilde{p}\) represent the actual emotion label and the predicted logical value respectively.