Abstract:Document sketching using Jaccard similarity has been a workable effective technique in reducing near-duplicates in Web page and image search results, and has also proven useful in file system synchronization, compression and learning applications. Min-wise sampling can be used to derive an unbiased estimator for Jaccard similarity and taking a few hundred independent consistent samples leads to compact sketches which provide good estimates of pairwise-similarity. Subsequent works extended this technique to weighted sets and show how to produce samples with only a constant number of hash evaluations for any element, independent of its weight. Another improvement by Li et al. shows how to speedup sketch computations by computing many (near-)independent samples in one shot. Unfortunately this latter improvement works only for the unweighted case. In this paper we give a simple, fast and accurate procedure which reduces weighted sets to unweighted sets with small impact on the Jaccard similarity. This leads to compact sketches consisting of many (near-)independent weighted samples which can be computed with just a small constant number of hash function evaluations per weighted element. The size of the produced unweighted set is furthermore a tunable parameter which enables us to run the unweighted scheme of Li et al. in the regime where it is most efficient. Even when the sets involved are unweighted, our approach gives a simple solution to the densification problem that other works attempted to address. Unlike previously known schemes, ours does not result in an unbiased estimator. However, we prove that the bias introduced by our reduction is negligible and that the standard deviation is comparable to the unweighted case. We also empirically evaluate our scheme and show that it gives significant gains in computational efficiency, without any measurable loss in accuracy.

Weight Evaluation for Features Via Constrained Data-Pairscan't-linkq

Similarity Calculation Based on Feature Weight Evaluation

Combination Weighting Approach Based on Deviation Function and Joint-Entropy

A novel method of constrained feature selection by the measurement of pairwise constraints uncertainty

A Probability-Based Multi-Measure Feature Matching Method in Map Conflation

Similarity Search Method Based on Weighted Vector Distance

Using Link-Based Content Analysis to Measure Document Similarity Effectively

Dynamic Weight-Based Individual Similarity Calculation for Information Searching in Social Computing

A Method of Weight Computing in the Comprehensive Estimation

Constraint Score: A new filter method for feature selection with pairwise constraints

Research on Semantic Similarity Calculation of Linked Data Based on Multiple Factors

An Improved Structural Equivalence Weighted Similarity for Recommender Systems

Hsim: A Novel Method on Similarity Computation by Hybrid Measure

SentPWNet: A Unified Sentence Pair Weighting Network for Task-specific Sentence Embedding

An attribute-weighted isometric embedding method for categorical encoding on mixed data

An Adaptive Method for the Efficient Similarity Calculation

Measuring Similarity Based on Link Information: A Comparative Study

Consistent Weighted Sampling Made Fast, Small, and Easy

Consistent Weighted Sampling Made More Practical

A Novel Similarity Measure Based on Weighted Bipartite Network for Collaborative Filtering Recommendation

Efficient Link-Based Similarity Search in Web Networks