Abstract:Over the past decade, wearable computing devices (``smart glasses'') have undergone remarkable advancements in sensor technology, design, and processing power, ushering in a new era of opportunity for high-density human behavior data. Equipped with wearable cameras, these glasses offer a unique opportunity to analyze non-verbal behavior in natural settings as individuals interact. Our focus lies in predicting engagement in dyadic interactions by scrutinizing verbal and non-verbal cues, aiming to detect signs of disinterest or confusion. Leveraging such analyses may revolutionize our understanding of human communication, foster more effective collaboration in professional environments, provide better mental health support through empathetic virtual interactions, and enhance accessibility for those with communication barriers. In this work, we collect a dataset featuring 34 participants engaged in casual dyadic conversations, each providing self-reported engagement ratings at the end of each conversation. We introduce a novel fusion strategy using Large Language Models (LLMs) to integrate multiple behavior modalities into a ``multimodal transcript'' that can be processed by an LLM for behavioral reasoning tasks. Remarkably, this method achieves performance comparable to established fusion techniques even in its preliminary implementation, indicating strong potential for further research and optimization. This fusion method is one of the first to approach ``reasoning'' about real-world human behavior through a language model. Smart glasses provide us the ability to unobtrusively gather high-density multimodal data on human behavior, paving the way for new approaches to understanding and improving human communication with the potential for important societal benefits. The features and data collected during the studies will be made publicly available to promote further research.

Consumer Intention Recognition and Behavior Prediction of Social E-commerce Users Based on Multimodal Fusion

Dynamical User Intention Prediction Via Multi-modal Learning

An Effective Multimodal Representation and Fusion Method for Multimodal Intent Recognition

A multimodal sentiment recognition method based on attention mechanism

An Efficient Multimodal Learning Framework to Comprehend Consumer Preferences Using BERT and Cross-Attention

Cross-Modal Sentiment Sensing with Visual-Augmented Representation and Diverse Decision Fusion

From Abstract to Details

Adaptive Sensitive Information Recognition Based on Multimodal Information Inference in Social Networks

Multimodal Data Guided Spatial Feature Fusion and Grouping Strategy for E-Commerce Commodity Demand Forecasting

Multi-Feature Fusion Multi-Modal Sentiment Analysis Model Based on Cross-Attention Mechanism

A Multimodal Late Fusion Model for E-Commerce Product Classification

Multimodal Sentiment Analysis Using Multi-tensor Fusion Network with Cross-modal Modeling

Multimodal Emotion Recognition Based on Feature Fusion.

Feature Extraction Network with Attention Mechanism for Data Enhancement and Recombination Fusion for Multimodal Sentiment Analysis

Multimodal sentiment analysis based on multiple attention

Multimodal Fusion with LLMs for Engagement Prediction in Natural Conversation

A Multi-Modal ELMo Model for Image Sentiment Recognition of Consumer Data

User Profiling through Deep Multimodal Fusion.

Multimodal Fusion Method Based on Self-Attention Mechanism

Multimodal Sentiment Analysis in Realistic Environments Based on Cross-Modal Hierarchical Fusion Network

A Multimodal Sentiment Analysis Method Integrating Multi-Layer Attention Interaction and Multi-Feature Enhancement