Abstract:With the emergence of social networking services, researchers enjoy the increasing availability of large-scale heterogenous datasets capturing online user interactions and behaviors. Traditional analysis of techno-social systems data has focused mainly on describing either the dynamics of social interactions, or the attributes and behaviors of the users. However, overwhelming empirical evidence suggests that the two dimensions affect one another, and therefore they should be jointly modeled and analyzed in a multi-modal framework. The benefits of such an approach include the ability to build better predictive models, leveraging social network information as well as user behavioral signals. To this purpose, here we propose the Constrained Latent Space Model (CLSM), a generalized framework that combines Mixed Membership Stochastic Blockmodels (MMSB) and Latent Dirichlet Allocation (LDA) incorporating a constraint that forces the latent space to concurrently describe the multiple data modalities. We derive an efficient inference algorithm based on Variational Expectation Maximization that has a computational cost linear in the size of the network, thus making it feasible to analyze massive social datasets. We validate the proposed framework on two problems: prediction of social interactions from user attributes and behaviors, and behavior prediction exploiting network information. We perform experiments with a variety of multi-modal social systems, spanning location-based social networks (Gowalla), social media services (Instagram, Orkut), e-commerce and review sites (Amazon, Ciao), and finally citation networks (Cora). The results indicate significant improvement in prediction accuracy over state of the art methods, and demonstrate the flexibility of the proposed approach for addressing a variety of different learning problems commonly occurring with multi-modal social data.

Learning Multimodal Latent Attributes

Attribute learning for understanding unstructured social activity

Attribute Learning for Image/Video Understanding

Learning User Attributes Via Mobile Social Multimedia Analytics

Joint Learning Of Semantic And Latent Attributes

Joint Semantic and Latent Attribute Modelling for Cross-Class Transfer Learning

Learning Explicit and Implicit Latent Common Spaces for Audio-Visual Cross-Modal Retrieval

Latent Semantic Factorization for Multimedia Representation Learning

Learning Tags from Unsegmented Videos of Multiple Human Actions.

Efficient Probabilistic Latent Semantic Analysis with Sparsity Control

Multimodal learning in loosely-organized web images

Visual Attributes Based Sparse Multitask Action Recognition

A Unified Multiplicative Framework for Attribute Learning

A Deep Approach For Multi-Modal User Attribute Modeling

Latent Space Model for Multi-Modal Social Data

Learning Subjective Attributes of Images from Auxiliary Sources

Beyond Semantic Attributes: Discrete Latent Attributes Learning for Zero-Shot Recognition

Multimodal few-shot classification without attribute embedding

Deep Attribute-preserving Metric Learning for Natural Language Object Retrieval

Learning Discriminative Latent Attributes for Zero-Shot Classification.