Abstract:With the continuous development of the Internet and information technology, more and more mobile terminals, wear equipment etc. contribute to the tremendous data. Thanks to the distributed computing, we can analyze the big data with quite high speed. However, many kinds of big data have an obvious common character that the datasets grow incrementally overtime, Which means the distributed computing should focus on incremental processing. A number of systems for incremental data processing are available, such as Google's Percolator and Yahoo's CBP. However, in order to utilize these mature framework, one needs to make a troublesome change for their program to adapt to the environment requirement. In this paper, we introduce a MapReduce framework, named Hadlnc, for efficient incremental computations. Hadlnc is designed for offline scenes, in which real-time is needless and in-memory cluster computing is invalid. Hadlnc takes the advantages of finer-grained computing and Content-defined Chunking(CDC) to make sure that the system can still reuse the results which we have computed before, even if the split data has been changed seriously. Instead of re-computing the changed data entirely, Hadlnc can quickly find out the difference between the new split and the old one, and then merge the delta and old results into the latest result of the new datasets. Meanwhile, the dividing stability of the datasets is a key factor for reusing the results. In order to guarantee the stability of the dataset's division, we propose a series of novel algorithms based on CDC. We implemented Hadlnc by extending the Hadoop framework, and evaluated it with many experiments including three specific cases and a practical case. From the comparing results it can be seen that the proposed Hadlnc is very efficient. (C) 2017 Elsevier B.V. All rights reserved.

A MapReduce based distributed LSI

A MapReduce Based Distributed LSI for Scalable Information Retrieval.

Document Clustering Using Locality Preserving Indexing

PPLSA: Parallel Probabilistic Latent Semantic Analysis Based on MapReduce.

SES-LSH: Shuffle-Efficient Locality Sensitive Hashing for Distributed Similarity Search

Big Data Processing with Probabilistic Latent Semantic Analysis on MapReduce

LI-MR: A Local Iteration Map/Reduce Model and Its Application to Mine Community Structure in Large-Scale Networks

Learning-based distributed locality sensitive hashing.

High-Dimensional Distributed Indexing Based on Locality-Sensitive Hashing

Distributed Index Mechanism Based on Hadoop.

Parallel Simrank Computing on Large Scale Dataset on Mapreduce

Distributed data management using MapReduce

Efficient Finer-Grained Incremental Processing with MapReduce for Big Data

Exploring Computation Locality of Graph Mining Algorithms on MapReduce

P2LSA and P2LSA+: Two Paralleled Probabilistic Latent Semantic Analysis Algorithms Based on the MapReduce Model.

Distributed Affinity Propagation Clustering Based on MapReduce

On Building and Updating Distributed Lsi for P2p Systems

Multidimensional Latent Semantic Analysis Using Term Spatial Information

LLM$\times$MapReduce: Simplified Long-Sequence Processing using Large Language Models

Locality Preserving Indexing for Document Representation.

A Simplified Latent Semantic Indexing Approach for Multi-Linguistic Information Retrieval.