Abstract:In the information explosion era, large scale data processing and mining is a hot issue. As microblog grows more popular, microblog services have become information provider on a web scale, so researches on microblog begin to focus more on its content mining than solely user's relationship analysis before. Although traditional text mining methods have been studied well, no algorithm is designed specially for microblog data, which contain structured information on social network besides plain text. In this paper, we introduce a novel probabilistic generative model MicroBlog-Latent Dirichlet Allocation (MB-LDA), which takes both contactor relevance relation and document relevance relation into consideration to improve topic mining in microblogs. Through Gibbs sampling for approximate inference of our model, MB-LDA can discover not only the topics of microblogs, but also the topics focused by contactors. When faced with large datasets, traditional techniques on single node become less practical within limited resources. So we present distributed MB-LDA in MapReduce framework in order to process large scale microblogs with high scalability. Furthermore, we apply a performance model to optimize the execution time by tuning the number of mappers and reducers. Experimental results on actual dataset show MB-LDA outperforms the baseline of LDA and distributed MB-LDA offers an effective solution to topic mining for large scale microblogs.

ACT-LDA: A Probabilistic Model of Topic, Community and User Influence

Topic Discovery Based on LDA_col Model and Topic Significance Re-ranking.

LDA-LPA: A Community Detection Method Based on Topic Model

Community-based Topic Modeling for Social Tagging.

A Local Context-Aware LDA Model for Topic Modeling in a Document Network

Large scale microblog mining using distributed MB-LDA.

Modeling Topic and Community Structure in Social Tagging: the TTR-LDA-Community Model.

News Topic Discovery Through Community Detection

Latent Dirichlet allocation (LDA) and topic modeling: models, applications, a survey

Probabilistic model for academic social network and its applications

Topic Analysis and Influential Paper Discovery on Scientific Publications

A Relationship Strength-Aware Topic Model for Communities Discovery in Online Social Networks

CLDA: an Effective Topic Model for Mining User Interest Preference under Big Data Background

MLLDA: Multi-level LDA for Modelling Users on Content Curation Social Networks

Clust-LDA: Joint Model for Text Mining and Author Group Inference

Latent Dirichlet Allocation - An approach for topic discovery

LDAExplore: Visualizing Topic Models Generated Using Latent Dirichlet Allocation

Towards Understanding Community Interests with Topic Modeling

Mining topic-level influence in heterogeneous networks.

Adding Community and Dynamic to Topic Models.

MR-LDA: An Efficient Topic Model for Classification of Short Text in Big Social Data.