Abstract:Recent years have witnessed the booming of online social media platforms with embracing the popular service called “Time-Sync Comment”, which supports the viewers to share their time-sync opinions along with video content. In this way, we observe that numerous semantically-altered terms, or “Memes”, were created by niche users to express their unique ideas and emotions, and further attracted a large group of viewers with better activity and enthusiasm. Unfortunately, since the memes were created based on domain-specific knowledge and semantically varied depending on the multimodal context in videos, newcomers may fail to comprehend the semantic connotation of memes, which may severely impair their user-experiences. To deal with this issue, in this article, we propose a novel meme explanation framework, called ProMDE, to automatically capture and comprehend the memes in time-sync comments, which could further benefit the viewers with meme explanation service. Specifically, we first iteratively reconstruct the original time-sync comments compared with visual embedding to detect the semantically-altered terms as meme candidates. Afterward, based on the guides from the domain-specific corpus, visual and textual features will be fused to represent the context-aware multimodal cues. Moreover, to accurately describe the commonly-seen homophones in memes, i.e., they have the same pronunciation but different word-spelling expressions, we integrate the phonetic symbols as an additional modality to enhance the framework. Finally, we utilize a Transformer-based decoder to generate the natural language explanation for captured memes. Extensive experiments on a large real-world dataset prove that our framework could significantly outperform several state-of-the-art baseline methods, demonstrating the efficacy of modeling multimodal context and pronunciation for meme detection and explanation.

Uncovering the Spatio-Temporal Dynamics of Memes in the Presence of Incomplete Information.

Two Ways of Understanding Social Dynamics: Analyzing the Predictability of Emergence of Objects in Reddit r/place Dependent on Locality in Space and Time

Location Inference for Non-geotagged Tweets in User Timelines

Contextualizing Internet Memes Across Social Media Platforms

Uncovering the Spatiotemporal Patterns of Collective Social Activity

MemeSequencer: Sparse Matching for Embedding Image Macros

Exploring Multi-Scale Spatiotemporal Twitter User Mobility Patterns with A Visual-Analytics Approach

Embedding and Predicting the Event at Early Stage

A Real-Time Method to Predict Social Media Popularity

Community Specific Temporal Topic Discovery from Social Media

Tracking Large-Scale Video Remix in Real-World Events

HelloMeme: Integrating Spatial Knitting Attentions to Embed High-Level and Fidelity-Rich Conditions in Diffusion Models

MOMENTA: A Multimodal Framework for Detecting Harmful Memes and Their Targets

Clustering memes in social media streams

Comprehending the Gossips: Meme Explanation in Time-Sync Video Comment via Multimodal Cues

Language agnostic meme-filtering for hashtag-based social network analysis

Meme Sentiment Analysis Enhanced with Multimodal Spatial Encoding and Facial Embedding

A Scalable Framework for Spatiotemporal Analysis of Location-based Social Media Data

General Geospatial Inference with a Population Dynamics Foundation Model

Transcending the Attention Paradigm: Representation Learning from Geospatial Social Media Data

Spatio-Temporal Turbulence Mitigation: A Translational Perspective