Abstract:Text clustering is a widely studied problem in the text mining domain. The Dirichlet Multinomial Mixture (DMM) model based clustering algorithms have shown good performance to cope with high dimensional sparse text data, obtaining reasonable results in both clustering accuracy and computational efficiency. However, the time complexity of DMM model training is proportional to the average document length and the number of clusters, making it inefficient for scaling up to long text and large corpora, which is common in realworld applications such as documents organization, retrieval and recommendation. In this paper, we leverage a symmetric prior setting for Dirichlet distribution, and build indices to decrease the time complexity of the sampling-based training for DMM from O(K∗L) to O(K∗U), where K is the number of clusters, L the average length of document, and U the average number of unique words in each document. We introduce a Metropolis-Hastings sampling algorithm, which further reduces the sampling time complexity from O(K∗U) to O(U) in the nearly-to-convergence training stages. Moreover, we also parallelize the DMM model training to obtain a further acceleration by using an uncollapsed Gibbs sampler. We combine all these optimizations into a highly efficient implementation, called X-DMM, which enables the DMM model to scale up for long and large-scale text clustering. We evaluate the performance of X-DMM on several real world datasets, and the experimental results show that XDMM achieves substantial speed up compared with existing state-of-the-art algorithms without clustering accuracy degradation.

Large-Scale Text Clustering Based on Improved K-Means Algorithm in the Storm Platform

Application Of Clustering Technologies In Very Large Power Grid Dispatch And Control System

Real-Time Spatial Queries for Moving Objects Using Storm Topology.

Large-scale Data Mining Method based on Clustering Algorithm Combined with MAPREDUCE

Big Data Real-Time Processing Based on Storm

Communication-Efficient Task Scheduling for Real-Time Distributed Computing.

Clustering Text Data Streams

An Improved K-means Algorithm Based on Mapreduce and Grid

Large-Scale Automatic K-Means Clustering for Heterogeneous Many-Core Supercomputer

Vhadoop: A Scalable Hadoop Virtual Cluster Platform for MapReduce-Based Parallel Machine Learning with Performance Consideration

Clustering Algorithm on Block Division of Documents

Algorithm and Experiment Research of Textual Document Clustering Based on Improved K-means

Clustering Massive Text Data Streams by Semantic Smoothing Model

An Efficient K-Means Clustering Algorithm On Mapreduce

An Improved K-Means Algorithm for Documents Clustering

A Linguistic Feature Based Text Clustering Method.

Hierarchical K-means Method for Clustering Large-Scale Advanced Metering Infrastructure Data

X-DMM: Fast and Scalable Model Based Text Clustering

A multi-threaded particle swarm optimization-kmeans algorithm based on MapReduce

Research and implementation of user clustering based on MapReduce in multimedia big data

HClustream: A Novel Approach for Clustering Evolving Heterogeneous Data Stream