Abstract:To find near-duplicate documents, fingerprint-based paradigms such as Broder's shingling and Charikar's simhash algorithms have been recognized as effective approaches and are considered the state-of-the-art. Nevertheless, we see two aspects of these approaches which may be improved. First, high score under these algorithms' similarity measurement implies high probability of similarity between documents, which is different from high similarity of the documents. But how similar two documents are is what we really need to know. Second, there has to be a tradeoff between hash-code length and hash-code multiplicity in fingerprint paradigms, which makes it hard to maintain a satisfactory recall level while improving precision. In this paper our contributions are two-folded. First, we propose a framework for implementing the longest common subsequence (LCS) as a similarity measurement in reasonable computing time, which leads to both high precision and recall. Second, we present an algorithm to get a trustable partition from the LCS to reduce the negative impact from templates used in web page design. A comprehensive experiment was conducted to evaluate our method in terms of its effectiveness, efficiency, and quality of result. More specifically, the method has been successfully used to partition a set of 430 million web pages into 68 million subsets of similar pages, which demonstrates its effectiveness. For quality, we compared our method with simhash and a Cosine-based method through a sampling process (Cosine is compared to LCS as an alternative similarity measurement). The result showed that our algorithm reached an overall precision of 0.95 while simhash was 0.71 and Cosine was 0.82. At the same time our method obtains 1.86 times as much recall as simhash and 1.56 times as much recall as Cosine. Comparison experiment was also done for documents in the same web sites. For that, our algorithm, simhash and Cosine find almost the same number of true-positives at a precision of 0.91, 0.50 and 0.63 respectively. In terms of efficiency, our algorithm takes 118 hours to process the whole archive of 430 million topic-type pages on a cluster of six Linux boxes, at the same time the processing time of simhash and Cosine is 94 hours and 68 hours respectively. When considering the need of word segmentation for languages such as Chinese, the processing time of Cosine should be multiplied and in our experiment it is 602 hours.

RETRACTED: NMLCS: A Novel Parallel Algorithm for Searching for the Multiple Longest Common Subsequences (Retracted Article)

Efficient Algorithms for Finding a Longest Common Increasing Subsequence

dwMLCS: An Efficient MLCS Algorithm based on Dynamic and Weighted Directed Acyclic Graph

A Space-Bounded Anytime Algorithm for the Multiple Longest Common Subsequence Problem

Projected Stream Water Fluxes of NO3 and Total Organic Carbon from the Storgama Headwater Catchment, Norway, under Climate Change and Reduced Acid Deposition

A Fast Longest Common Subsequence Algorithm for Biosequences Alignment

An algorithm for rapid noncoding RNA sequence-structure alignment

Polynomial-time equivalences and refined algorithms for longest common subsequence variants

An efficient parallel algorithm for multiple sequence similarities calculation using a low complexity method.

Parallel Computing for the Longest Common Subsequences in Network Intrusion Detection System

Algorithms for the Uniqueness of the Longest Common Subsequence

The aliens among us: How invasive species are transforming the planet—And ourselves Yale University Press, New Haven Connecticut, 2017, xii + 353 pp, ISBN 978-0-300-20890-0

Efficient algorithms for the longest common subsequence in $k$-length substrings

Retracted: Design and Implementation of Online Japanese Examination System Based on Genetic Algorithm

Accelerating spliced alignment of long RNA sequencing reads using parallel maximal exact match retrieval

Multiple Sequence Alignment based on deep Q network with negative feedback policy

Retracted: Multiple-Devices-Process Integrated Scheduling Algorithm with Time-Selective Strategy for Process Sequence

Retracted: Design of English Translation Model Based on Recurrent Neural Network

Achieving Both High Precision and High Recall in Near-Duplicate Detection.

Algorithms For Loosely Constrained Multiple Sequence Alignment