Abstract:Laughter is considered one of the most overt signals of joy. Laughter is well-recognized as a multimodal phenomenon but is most commonly detected by sensing the sound of laughter. It is unclear how perception and annotation of laughter differ when annotated from other modalities like video, via the body movements of laughter. In this paper we take a first step in this direction by asking if and how well laughter can be annotated when only audio, only video (containing full body movement information) or audiovisual modalities are available to annotators. We ask whether annotations of laughter are congruent across modalities, and compare the effect that labeling modality has on machine learning model performance. We compare annotations and models for laughter detection, intensity estimation, and segmentation, three tasks common in previous studies of laughter. Our analysis of more than 4000 annotations acquired from 48 annotators revealed evidence for incongruity in the perception of laughter, and its intensity between modalities. Further analysis of annotations against consolidated audiovisual reference annotations revealed that recall was lower on average for video when compared to the audio condition, but tended to increase with the intensity of the laughter samples. Our machine learning experiments compared the performance of state-of-the-art unimodal (audio-based, video-based and acceleration-based) and multi-modal models for different combinations of input modalities, training label modality, and testing label modality. Models with video and acceleration inputs had similar performance regardless of training label modality, suggesting that it may be entirely appropriate to train models for laughter detection from body movements using video-acquired labels, despite their lower inter-rater agreement.

An Analysis of Rhythmic Staccato-Vocalization Based on Frequency Demodulation for Laughter Detection in Conversational Meetings

Happy or Evil Laughter? Analysing a Database of Natural Audio Samples

Acoustic analysis of several laughter types in conversational dialogues

Analysis of Co-Laughter Gesture Relationship on RGB videos in Dyadic Conversation Contex

Impact of annotation modality on label quality and model performance in the automatic assessment of laughter in-the-wild

Laugh Betrays You? Learning Robust Speaker Representation From Speech Containing Non-Verbal Fragments

ha-HA-hha? Intensity and voice quality characteristics of laughter

On Laughter and Speech-Laugh, Based on Observations of Child-Robot Interaction

A New Perspective on Smiling and Laughter Detection: Intensity Levels Matter

Analyzing long-term rhythm variations in Mising and Assamese using frequency domain correlates

Haha-Pod: An Attempt for Laughter-based Non-Verbal Speaker Verification

Laughter in everyday life: an event-based experience sampling method study using wrist-worn wearables

LaughNet: synthesizing laughter utterances from waveform silhouettes and a single laughter example

Detecting Affiliation in Colaughter Across 24 Societies.

Generating Diverse Realistic Laughter for Interactive Art

Making Flow-Matching-Based Zero-Shot Text-to-Speech Laugh as You Like

Heterogeneous Mixture Models Using Sparse Representation Features For Applause And Laugh Detection

LaughTalk: Expressive 3D Talking Head Generation with Laughter

Executive Voiced Laughter and Social Approval: An Explorative Machine Learning Study

Laughter Synthesis using Pseudo Phonetic Tokens with a Large-scale In-the-wild Laughter Corpus

Use of Laughter for the Detection of Parkinson's Disease: Feasibility Study for Clinical Decision Support Systems, Based on Speech Recognition and Automatic Classification Techniques