Abstract:Gestures are inherent to human interaction and often complement speech in face-to-face communication, forming a multimodal communication system. An important task in gesture analysis is detecting a gesture's beginning and end. Research on automatic gesture detection has primarily focused on visual and kinematic information to detect a limited set of isolated or silent gestures with low variability, neglecting the integration of speech and vision signals to detect gestures that co-occur with speech. This work addresses this gap by focusing on co-speech gesture detection, emphasising the synchrony between speech and co-speech hand gestures. We address three main challenges: the variability of gesture forms, the temporal misalignment between gesture and speech onsets, and differences in sampling rate between modalities. We investigate extended speech time windows and employ separate backbone models for each modality to address the temporal misalignment and sampling rate differences. We utilize Transformer encoders in cross-modal and early fusion techniques to effectively align and integrate speech and skeletal sequences. The study results show that combining visual and speech information significantly enhances gesture detection performance. Our findings indicate that expanding the speech buffer beyond visual time segments improves performance and that multimodal integration using cross-modal and early fusion techniques outperforms baseline methods using unimodal and late fusion methods. Additionally, we find a correlation between the models' gesture prediction confidence and low-level speech frequency features potentially associated with gestures. Overall, the study provides a better understanding and detection methods for co-speech gestures, facilitating the analysis of multimodal communication.

Extending the Architecture of Language From a Multimodal Perspective

A multimodal parallel architecture: A cognitive framework for multimodal interactions

Multimodal Distributional Semantics

Beyond robotic speech: mutual benefits to cognitive psychology and artificial intelligence from the study of multimodal communication

Multimodal Grounding for Language Processing

There exist consistent temporal logics admitting changes of History

Multimodal research in vision and language: A review of current and emerging trends

Leveraging Speech for Gesture Detection in Multimodal Communication

Speakers’ Gestures and Semantic Analysis

Multimodal Language Specification for Human Adaptive Mechatronics

A modular architecture for creating multimodal agents

Developing Embodied Multisensory Dialogue Agents

Breaking Boundaries Between Linguistics and Artificial Intelligence

With Ears to See and Eyes to Hear: Sound Symbolism Experiments with Multimodal Large Language Models

Entity Embeddings : Perspectives Towards an Omni-Modality Era for Large Language Models

Multimodal Large Language Models: A Survey

Towards Multimodal Content Representation

Semiotically-grounded distant viewing of diagrams: insights from two multimodal corpora

Language as Description, Indication, and Depiction

A multimodal study of augmented reality in the architectural design studio