Abstract:With the emergence of the big data age, methods for quickly and accurately obtaining valuable hot topics from the vast amount of digitized textual material have attracted much attention. In this work, we focus on topic detection in microblogs in the big data environment. Different from existing approaches, we solve this problem in a distributed way. Specifically, we propose a non-iterative algorithm called parallel two-phase mic-mac hot topic detection (TMHTD), and implement it in the Apache Spark environment. The proposed TMHTD method includes two phases, i.e., the micro-clustering phase and the macro-clustering phase. To improve the accuracy of hot topic detection, three optimization methods, along with TMHTD, are proposed. To handle large databases, we deliberately design a group of MapReduce jobs to concretely accomplish hot topic detection in a highly scalable way. We compare the TMHTD algorithm with the general single-pass algorithm and the Latent Dirichlet Allocation (LDA) algorithm. Our experiments are carried out on real-life data sets gathered from the Sina Weibo API. Extensive experimental results indicate that the accuracy and performance of the TMHTD algorithm are significant improvements over previous methods. More specifically, the F-measure value of the TMHTD algorithm shows a 6% and 8% improvement over the general single-pass algorithm and the LDA algorithm, respectively. The run time of the TMHTD algorithm is 7 times and twice as superior to the general single-pass algorithm and the LDA algorithm, respectively. (C) 2017 Published by Elsevier B.V.

Detecting Hot Topics in Technology News Streams

Detecting hot topics in chinese microblog streams based on frequent patterns mining

Hot Topic Detection Based on a Refined TF-IDF Algorithm

Hot topic detection based on a tripartite model

Online Hot Topic Detection from Web News Based on Bursty Term Identification.

Detection and Extraction of Hot Topics on Chinese Microblogs

Research on Topic Discovery Technology for Web News.

An Effective Hot Topic Detection Method for Microblog on Spark

An Efficient Algorithm of Hot Events Detection in Text Streams

Hot topic detection in news blogs from the perspective of w2t

Bursty Feature Based Topic Detction and Summarization

Research on Topic Detection and Tracking for Online News Texts

Design and Implementation of On-Line Hot Topic Discovery Model

Research And Implementation Of Hot Topic Detection System Based On Web

Micro-blogging Hot Words Extraction and Topic Detection

Short Texts'Hot Topics Detection: Based on Word Frequency Mean Fluctuation and Probabilistic Language Model

Detecting Hot Topics from Academic Big Data

Hot topic detection based on combined content and time similarity

Data Mining Method of Social Media Hot Topics Based on Time Series Clustering

Approach to Extracting Hot Topics Based on Network Traffic Content

News topic detection based on hierarchical clustering and named entity.