Abstract:In the information explosion era, large scale data processing and mining is a hot issue. As microblog grows more popular, microblog services have become information provider on a web scale, so researches on microblog begin to focus more on its content mining than solely user's relationship analysis before. Although traditional text mining methods have been studied well, no algorithm is designed specially for microblog data, which contain structured information on social network besides plain text. In this paper, we introduce a novel probabilistic generative model MicroBlog-Latent Dirichlet Allocation (MB-LDA), which takes both contactor relevance relation and document relevance relation into consideration to improve topic mining in microblogs. Through Gibbs sampling for approximate inference of our model, MB-LDA can discover not only the topics of microblogs, but also the topics focused by contactors. When faced with large datasets, traditional techniques on single node become less practical within limited resources. So we present distributed MB-LDA in MapReduce framework in order to process large scale microblogs with high scalability. Furthermore, we apply a performance model to optimize the execution time by tuning the number of mappers and reducers. Experimental results on actual dataset show MB-LDA outperforms the baseline of LDA and distributed MB-LDA offers an effective solution to topic mining for large scale microblogs.

News Topic Discovery Through Community Detection

Topic Discovery Based on LDA_col Model and Topic Significance Re-ranking.

Query Subtopic Mining Via Subtractive Initialization of Non-negative Sparse Latent Semantic Analysis

Mining Coherent Topics in Documents Using Word Embeddings and Large-Scale Text Data

A Knowledge-Based Semisupervised Hierarchical Online Topic Detection Framework.

A Survey on News Text Visualization via Probabilistic Topic Modeling

ACT-LDA: A Probabilistic Model of Topic, Community and User Influence

Topic Modeling on Online News.Portal Using Latent Dirichlet Allocation (LDA)

Large scale microblog mining using distributed MB-LDA.

Community-based Topic Modeling for Social Tagging.

Topic2Vec: Learning distributed representations of topics

Contextual-LDA: A Context Coherent Latent Topic Model for Mining Large Corpora.

A Fast Hierarchical Topic Detection Method

Constrained-hLDA for Topic Discovery in Chinese Microblogs.

Latent Dirichlet allocation (LDA) and topic modeling: models, applications, a survey

Community Discovery Algorithm Based on User Behavior Similarity

Research on Mining Common Concern Via Infinite Topic Modelling

Term-community-based topic detection with variable resolution

Quantitative analysis of large amounts of journalistic texts using topic modelling

The Discovery of Burst Topic and Its Intermittent Evolution in Our Real World

Detecting Public Influence on News Using Topic-Aware Dynamic Granger Test.