Abstract:There has been an explosion of multimodal content generated on social media networks in the last few years, which has necessitated a deeper understanding of social media content and user behavior. We present a novel content-independent content-user-reaction model for social multimedia content analysis. Compared to prior works that generally tackle semantic content understanding and user behavior modeling in isolation, we propose a generalized solution to these problems within a unified framework. We embed users, images and text drawn from open social media in a common multimodal geometric space, using a novel loss function designed to cope with distant and disparate modalities, and thereby enable seamless three-way retrieval. Our model not only outperforms unimodal embedding based methods on cross-modal retrieval tasks but also shows improvements stemming from jointly solving the two tasks on Twitter data. We also show that the user embeddings learned within our joint multimodal embedding model are better at predicting user interests compared to those learned with unimodal content on Instagram data. Our framework thus goes beyond the prior practice of using explicit leader-follower link information to establish affiliations by extracting implicit content-centric affiliations from isolated users. We provide qualitative results to show that the user clusters emerging from learned embeddings have consistent semantics and the ability of our model to discover fine-grained semantics from noisy and unstructured data. Our work reveals that social multimodal content is inherently multimodal and possesses a consistent structure because in social networks meaning is created through interactions between users and content.

Deep Collaborative Embedding for Social Image Understanding.

Weakly Supervised Deep Matrix Factorization for Social Image Understanding.

Dual Collaborative Visual-Semantic Mapping for Multi-Label Zero-Shot Image Recognition

Learning from collective intelligence: Feature learning using social images and tags

Deep Matrix Factorization for Social Image Tag Refinement and Assignment.

Deep Multi-View Representation Learning for Social Images

Collaborative Feature Learning from Social Media

Collaborative Image Understanding

Deep Attentive Multimodal Network Representation Learning for Social Media Images

Learning Deep Structure-Preserving Image-Text Embeddings

Stacked Autoencoder Based Weak Supervision for Social Image Understanding.

Learning a Unified Embedding for Visual Search at Pinterest

Social-Guided Representation Learning for Images via Deep Heterogeneous Hypergraph Embedding.

From Content to Links: Social Image Embedding with Deep Multimodal Model.

Deep Unified Multimodal Embeddings for Understanding both Content and Users in Social Media Networks

Deep Collaborative Discrete Hashing With Semantic-Invariant Structure Construction

Cross-Modal Image-Tag Relevance Learning for Social Images

Visual Understanding by Mining Social Media: Recent Advances and Challenges

Learning Social Image Embedding with Deep Multimodal Attention Networks

Learning Semantic Embedding At A Large Scale

Collaboration Analysis Using Deep Learning