Abstract:Recent years have witnessed the booming of online social media platforms with embracing the popular service called “Time-Sync Comment”, which supports the viewers to share their time-sync opinions along with video content. In this way, we observe that numerous semantically-altered terms, or “Memes”, were created by niche users to express their unique ideas and emotions, and further attracted a large group of viewers with better activity and enthusiasm. Unfortunately, since the memes were created based on domain-specific knowledge and semantically varied depending on the multimodal context in videos, newcomers may fail to comprehend the semantic connotation of memes, which may severely impair their user-experiences. To deal with this issue, in this article, we propose a novel meme explanation framework, called ProMDE, to automatically capture and comprehend the memes in time-sync comments, which could further benefit the viewers with meme explanation service. Specifically, we first iteratively reconstruct the original time-sync comments compared with visual embedding to detect the semantically-altered terms as meme candidates. Afterward, based on the guides from the domain-specific corpus, visual and textual features will be fused to represent the context-aware multimodal cues. Moreover, to accurately describe the commonly-seen homophones in memes, i.e., they have the same pronunciation but different word-spelling expressions, we integrate the phonetic symbols as an additional modality to enhance the framework. Finally, we utilize a Transformer-based decoder to generate the natural language explanation for captured memes. Extensive experiments on a large real-world dataset prove that our framework could significantly outperform several state-of-the-art baseline methods, demonstrating the efficacy of modeling multimodal context and pronunciation for meme detection and explanation.

MemeFaceGenerator: Adversarial Synthesis of Chinese Meme-face from Natural Sentences

Expressive Facial Style Transfer for Personalized Memes Mimic

Automatic Chinese Meme Generation Using Deep Neural Networks

Multi-modal application: Image Memes Generation

Multimodal-driven Talking Face Generation, Face Swapping, Diffusion Model

MemeCraft: Contextual and Stance-Driven Multimodal Meme Generation

Multi-expression Generative Adversarial Networks for Facial Expression Synthesis

XMeCap: Meme Caption Generation with Sub-Image Adaptability

HelloMeme: Integrating Spatial Knitting Attentions to Embed High-Level and Fidelity-Rich Conditions in Diffusion Models

memeBot: Towards Automatic Image Meme Generation

Generating Multimodal Metaphorical Features for Meme Understanding

DMF-GAN: Deep Multimodal Fusion Generative Adversarial Networks for Text-to-Image Synthesis

Meme Sentiment Analysis Enhanced with Multimodal Spatial Encoding and Facial Embedding

CMAFGAN: A Cross-Modal Attention Fusion based Generative Adversarial Network for attribute word-to-face synthesis

FaceChain: A Playground for Identity-Preserving Portrait Generation

Comprehending the Gossips: Meme Explanation in Time-Sync Video Comment via Multimodal Cues

MemeFier: Dual-stage Modality Fusion for Image Meme Classification

Comp-GAN

Memeplate: A Chinese Multimodal Dataset for Humor Understanding in Meme Templates

Text-to-face Synthesis Based on Facial Landmarks Prediction

DM-GAN: Dynamic Memory Generative Adversarial Networks for Text-to-Image Synthesis