Abstract:In the information explosion era, large scale data processing and mining is a hot issue. As microblog grows more popular, microblog services have become information provider on a web scale, so researches on microblog begin to focus more on its content mining than solely user's relationship analysis before. Although traditional text mining methods have been studied well, no algorithm is designed specially for microblog data, which contain structured information on social network besides plain text. In this paper, we introduce a novel probabilistic generative model MicroBlog-Latent Dirichlet Allocation (MB-LDA), which takes both contactor relevance relation and document relevance relation into consideration to improve topic mining in microblogs. Through Gibbs sampling for approximate inference of our model, MB-LDA can discover not only the topics of microblogs, but also the topics focused by contactors. When faced with large datasets, traditional techniques on single node become less practical within limited resources. So we present distributed MB-LDA in MapReduce framework in order to process large scale microblogs with high scalability. Furthermore, we apply a performance model to optimize the execution time by tuning the number of mappers and reducers. Experimental results on actual dataset show MB-LDA outperforms the baseline of LDA and distributed MB-LDA offers an effective solution to topic mining for large scale microblogs.

Mining Parallel Corpus from Sina Microblog

Method of Sina microblogging big data grabbing based on multi-strategy and its application

Large scale microblog mining using distributed MB-LDA.

Multi-Strategy Based Sina Microblog Data Acquisition For Opinion Mining

Mining the Interests of Chinese Microbloggers Via Keyword Extraction

News Comments Generation Via Mining Microblogs.

Mining recipes in microblog

A System to Manage and Mine Microblogging Data.

A Japanese-Chinese Parallel Corpus Using Crowdsourcing for Web Mining

Mining Social Relationships in Micro-blogging Systems.

Compact Indexing and Judicious Searching for Billion-Scale Microblog Retrieval.

Mining the Personal Interests of Microbloggers via Exploiting Wikipedia Knowledge

Mining User's Real Social Circle in Microblog

Mining Concurrent Topical Activity in Microblog Streams

Real-Time Search over a Microblogging System

Collective Opinion Target Extraction in Chinese Microblogs.

Mining aspects and opinions from microblog events

MicroScholar: Mining Scholarly Information from Chinese Microblogs.

A comparative study of users' microblogging behavior on sina weibo and twitter

Micro-blogs Data Collection Based on MapReduce

Discovering Knowledge in Microblog Based on Naturally Annotated Web Resources