Abstract:Along with the fast evolvement of internet technology, internet users have to face the large amount of textual data every day. Apparently, organizing texts into categories can help users dig the useful information from large-scale text collection. Clustering is one of the most promising tools for categorizing texts due to its unsupervised characteristic. Unfortunately, most of traditional clustering algorithms lose their high qualities on large-scale text collection, which mainly attributes to the high-dimensional vector space and semantic similarity among texts. To effectively and efficiently cluster large-scale text collection, this paper puts forward a vector reconstruction based clustering algorithm. Only the features that can represent the cluster are preserved in cluster's representative vector. This algorithm alternately repeats two sub-processes until it converges. One process is partial tuning sub-process, where feature's weight is fine-tuned by iterative process similar to self-organizing-mapping (SOM) algorithm. To accelerate clustering velocity, an intersection based similarity measurement and its corresponding neuron adjustment function are proposed and implemented in this sub-process. The other process is overall tuning sub-process, where the features are reallocated among different clusters. In this sub-process, the features useless to represent the cluster are removed from cluster's representative vector. Experimental results on the three text collections (including two small-scale and one large-scale text collections) demonstrate that our algorithm obtains high-quality performances on both small-scale and large-scale text collections.

An Information Retrieval Algorithm for Massive and Real-time Data

Implementation of large-scale distributed information retrieval system

Research on the Private Customized Information Retrieval Based on Hadoop Cluster

A Real-Time C-V Clustering Algorithm for WEB-mining

Research of Massive Internet Text Data Real-Time Loading and Index System

The Clustering Analysis Technology for Information Retrieval

A Study of Performance Optimization Method for Massive Spaito-temporal Data Based on Spatio-temporal Partition Clustering

Massive-Scale Visual Information Retrieval towards City Residential Environment Surveillance

Distributed Information Theoretic Clustering

The Design and Implementation of an Intelligent Distributed Text Retrieval System

An efficient clustering algorithm for large-scale topical web pages.

CD-Tree: A Clustering-Based Dynamic Indexing and Retrieval Approach.

A Vector Reconstruction Based Clustering Algorithm Particularly for Large-Scale Text Collection

A Data Clustering Algorithm over Real Time High-Volume Data Streams

STATE OF THE ART OF INFORMATION RETRIEVAL ON THE WEB

VRCA: A Clustering Algorithm for Massive Amount of Texts

Large-Scale Network Traffic Analysis and Retrieval System Using CFS Algorithm

Research and Implementation of Multi-Terabyte Data Real-Time Loading Technology

Web Pages Clustering and Concepts Mining: an Approach Towards Intelligent Information Retrieval

Research Development of Clustering Search Engine

An Efficient Clustering Method For Retrieval Of Large Image Databases