Abstract:With the emergence of the big data age, methods for quickly and accurately obtaining valuable hot topics from the vast amount of digitized textual material have attracted much attention. In this work, we focus on topic detection in microblogs in the big data environment. Different from existing approaches, we solve this problem in a distributed way. Specifically, we propose a non-iterative algorithm called parallel two-phase mic-mac hot topic detection (TMHTD), and implement it in the Apache Spark environment. The proposed TMHTD method includes two phases, i.e., the micro-clustering phase and the macro-clustering phase. To improve the accuracy of hot topic detection, three optimization methods, along with TMHTD, are proposed. To handle large databases, we deliberately design a group of MapReduce jobs to concretely accomplish hot topic detection in a highly scalable way. We compare the TMHTD algorithm with the general single-pass algorithm and the Latent Dirichlet Allocation (LDA) algorithm. Our experiments are carried out on real-life data sets gathered from the Sina Weibo API. Extensive experimental results indicate that the accuracy and performance of the TMHTD algorithm are significant improvements over previous methods. More specifically, the F-measure value of the TMHTD algorithm shows a 6% and 8% improvement over the general single-pass algorithm and the LDA algorithm, respectively. The run time of the TMHTD algorithm is 7 times and twice as superior to the general single-pass algorithm and the LDA algorithm, respectively. (C) 2017 Published by Elsevier B.V.

Hotspots Detection on Microblog

Detecting Hotspot Information Using Multi-Attribute Based Topic Model.

A Research of Hot Topic Detection through Microblogging

An Effective Hot Topic Detection Method for Microblog on Spark

An Improved Latent Dirichlet Allocation Model for Hot Topic Extraction

Detection and Extraction of Hot Topics on Chinese Microblogs

Microblog Topic Detection Based on LDA Model and Multi-level Clustering

Micro-blogging Hot Words Extraction and Topic Detection

Hot Topic Detection Based on VSM and Improved LDA Hybrid Model

A Microblog Hot Topic Detection Algorithm Based on Discrete Particle Swarm Optimization

Microblog hotspot detection method based on MFIHC and TOPSIS

Large scale microblog mining using distributed MB-LDA.

Micro-blog Topic Detection in Frequent Word Networks

Microblog hot topic detection based on positive point mutual informa-tion and probabilistic topic model

Popular Topic Detection in Chinese Micro-Blog Based on the Modified LDA Model

Micro-blog topic detection based on users’ interests and communities

Detecting hot topics in chinese microblog streams based on frequent patterns mining

A Twitter hotspot mining method based on sematic clustering of word vectors

Hotspot Detection in Social Media Based on Improved Strategy Clustering

Research on User Influence-Based Hot Topic Detection Methodin Micro-blog

A Topic Detection Method for Chinese Microblog