Abstract:Stories can have tremendous power -- not only useful for entertainment, they can activate our interests and mobilize our actions. The degree to which a story resonates with its audience may be in part reflected in the emotional journey it takes the audience upon. In this paper, we use machine learning methods to construct emotional arcs in movies, calculate families of arcs, and demonstrate the ability for certain arcs to predict audience engagement. The system is applied to Hollywood films and high quality shorts found on the web. We begin by using deep convolutional neural networks for audio and visual sentiment analysis. These models are trained on both new and existing large-scale datasets, after which they can be used to compute separate audio and visual emotional arcs. We then crowdsource annotations for 30-second video clips extracted from highs and lows in the arcs in order to assess the micro-level precision of the system, with precision measured in terms of agreement in polarity between the system's predictions and annotators' ratings. These annotations are also used to combine the audio and visual predictions. Next, we look at macro-level characterizations of movies by investigating whether there exist `universal shapes' of emotional arcs. In particular, we develop a clustering approach to discover distinct classes of emotional arcs. Finally, we show on a sample corpus of short web videos that certain emotional arcs are statistically significant predictors of the number of comments a video receives. These results suggest that the emotional arcs learned by our approach successfully represent macroscopic aspects of a video story that drive audience engagement. Such machine understanding could be used to predict audience reactions to video stories, ultimately improving our ability as storytellers to communicate with each other.

OdorAgent: Generate Odor Sequences for Movies Based on Large Language Model

Odor Emoticon: an Olfactory Application That Conveys Emotions.

Aroma Release of Olfactory Displays Based on Audio-Visual Content

ScentClue: Enhancing Story Engagement in Virtual Reality Through Hedonically Varied Olfactory Hints

MovieFactory: Automatic Movie Creation from Text using Large Generative Models for Language and Images

Odor Prediction and Aroma Mixture Design Using Machine Learning Model and Molecular Surface Charge Density Profiles

Molecule Generation and Optimization for Efficient Fragrance Creation

OdorCarousel: A Design Tool for Customizing Smell-Enhanced Virtual Experiences

Visual Analysis of Odor Interaction Based on Support Vector Regression Method.

Insight into the Structure–Odor Relationship of Molecules: A Computational Study Based on Deep Learning

Advancing Odor Classification Models Enhanced by Scientific Machine Learning and Mechanistic Model: Probabilistic Weight Assignment for Odor Intensity Prediction and Uncertainty Analysis for Robust Fragrance Classification

Predicting human olfactory perception from chemical features of odor molecules

Generating Event-oriented Attribution for Movies via Two-Stage Prefix-Enhanced Multimodal LLM

Audio-Visual Sentiment Analysis for Learning Emotional Arcs in Movies

Understanding the Odour Spaces: A Step towards Solving Olfactory Stimulus-Percept Problem

Smell Impregnated Images, Encoding, Generation and Its Implications

Semantically consistent Video-to-Audio Generation using Multimodal Language Large Model

A Publicly Available Multimodal Emotional Dataset Triggered by Both Videos and Odors

An Investigation of Olfactory-Enhanced Video on EEG-Based Emotion Recognition

Exploring Potential Scenarios and Design Implications Through a Camera-like Physical Odor Capture Prototype.

OlfacEnhancer: A Vision-Based Scented Necklace for Cross-Modal Perception and Olfaction Augmentation