Abstract:Laughter is considered one of the most overt signals of joy. Laughter is well-recognized as a multimodal phenomenon but is most commonly detected by sensing the sound of laughter. It is unclear how perception and annotation of laughter differ when annotated from other modalities like video, via the body movements of laughter. In this paper we take a first step in this direction by asking if and how well laughter can be annotated when only audio, only video (containing full body movement information) or audiovisual modalities are available to annotators. We ask whether annotations of laughter are congruent across modalities, and compare the effect that labeling modality has on machine learning model performance. We compare annotations and models for laughter detection, intensity estimation, and segmentation, three tasks common in previous studies of laughter. Our analysis of more than 4000 annotations acquired from 48 annotators revealed evidence for incongruity in the perception of laughter, and its intensity between modalities. Further analysis of annotations against consolidated audiovisual reference annotations revealed that recall was lower on average for video when compared to the audio condition, but tended to increase with the intensity of the laughter samples. Our machine learning experiments compared the performance of state-of-the-art unimodal (audio-based, video-based and acceleration-based) and multi-modal models for different combinations of input modalities, training label modality, and testing label modality. Models with video and acceleration inputs had similar performance regardless of training label modality, suggesting that it may be entirely appropriate to train models for laughter detection from body movements using video-acquired labels, despite their lower inter-rater agreement.

ha-HA-hha? Intensity and voice quality characteristics of laughter

Acoustic analysis of several laughter types in conversational dialogues

The Perception of Spontaneous and Volitional Laughter Across 21 Societies.

Happy or Evil Laughter? Analysing a Database of Natural Audio Samples

An Analysis of Rhythmic Staccato-Vocalization Based on Frequency Demodulation for Laughter Detection in Conversational Meetings

LaughNet: synthesizing laughter utterances from waveform silhouettes and a single laughter example

Recognition of emotions in German laughter across cultures

Laughter and culture

Impact of annotation modality on label quality and model performance in the automatic assessment of laughter in-the-wild

A generative framework for conversational laughter: Its 'language model' and laughter sound synthesis

The relationship between familiarity, gender, disagreement, and status and bouts of solitary and joint laughter

The paradoxical role of emotional intensity in the perception of vocal affect

On Laughter and Speech-Laugh, Based on Observations of Child-Robot Interaction

Individual differences in laughter perception reveal roles for mentalizing and sensorimotor systems in the evaluation of emotional authenticity

The Effects of Loudness and Smiling on Timbre Features: Implications for Charismatic Voices in Mandarin, German and Danish.

I thought that I heard you laughing: Contextual facial expressions modulate the perception of authentic laughter and crying

Laugh Betrays You? Learning Robust Speaker Representation From Speech Containing Non-Verbal Fragments

Detecting Affiliation in Colaughter Across 24 Societies.

Haha-Pod: An Attempt for Laughter-based Non-Verbal Speaker Verification

Laughter Synthesis using Pseudo Phonetic Tokens with a Large-scale In-the-wild Laughter Corpus

Making Flow-Matching-Based Zero-Shot Text-to-Speech Laugh as You Like