Abstract:With the emergence of the big data age, methods for quickly and accurately obtaining valuable hot topics from the vast amount of digitized textual material have attracted much attention. In this work, we focus on topic detection in microblogs in the big data environment. Different from existing approaches, we solve this problem in a distributed way. Specifically, we propose a non-iterative algorithm called parallel two-phase mic-mac hot topic detection (TMHTD), and implement it in the Apache Spark environment. The proposed TMHTD method includes two phases, i.e., the micro-clustering phase and the macro-clustering phase. To improve the accuracy of hot topic detection, three optimization methods, along with TMHTD, are proposed. To handle large databases, we deliberately design a group of MapReduce jobs to concretely accomplish hot topic detection in a highly scalable way. We compare the TMHTD algorithm with the general single-pass algorithm and the Latent Dirichlet Allocation (LDA) algorithm. Our experiments are carried out on real-life data sets gathered from the Sina Weibo API. Extensive experimental results indicate that the accuracy and performance of the TMHTD algorithm are significant improvements over previous methods. More specifically, the F-measure value of the TMHTD algorithm shows a 6% and 8% improvement over the general single-pass algorithm and the LDA algorithm, respectively. The run time of the TMHTD algorithm is 7 times and twice as superior to the general single-pass algorithm and the LDA algorithm, respectively. (C) 2017 Published by Elsevier B.V.

A Research of Hot Topic Detection through Microblogging

Detection and Extraction of Hot Topics on Chinese Microblogs

Research on algorithm of extracting micro-blog's hot topics

Research on User Influence-Based Hot Topic Detection Methodin Micro-blog

Detecting hot topics in chinese microblog streams based on frequent patterns mining

Micro-blogging Hot Words Extraction and Topic Detection

Hotspots Detection on Microblog

An Effective Hot Topic Detection Method for Microblog on Spark

A Topic Detection Method for Chinese Microblog

Research on Microblog Hot Topic Detection Method Based on Term Energy Change

Hot Word Extraction for Microblog Based on Massive Data Filtering

A Microblog Hot Topic Detection Algorithm Based on Discrete Particle Swarm Optimization

A Popular Topic Detection Method based on Microblog Images and Short Text Information

Micro-blog topic detection based on users’ interests and communities

A Prerecognition Model for Hot Topic Discovery Based on Microblogging Data

Microblog hot topic detection based on positive point mutual informa-tion and probabilistic topic model

Hot Issues Detection on Weibo Based on Social Network Analysis

An Improved Latent Dirichlet Allocation Model for Hot Topic Extraction

News Topic Detection Approach on Chinese Microblog

Hot Topic Detection Based on VSM and Improved LDA Hybrid Model

Hot topic analysis and content mining in social media