Abstract:A number of tasks in classification, information retrieval, recommendation systems, and record linkage reduce to the core problem of inner product similarity join (IPS join): identifying pairs of vectors in a collection that have a sufficiently large inner product. IPS join is well understood when vectors are normalized and some approximation of inner products is allowed. However, the general case where vectors may have any length appears much more challenging. Recently, new upper bounds based on asymmetric locality-sensitive hashing (ALSH) and asymmetric embeddings have emerged, but little has been known on the lower bound side. In this paper we initiate a systematic study of inner product similarity join, showing new lower and upper bounds. Our main results are: * Approximation hardness of IPS join in subquadratic time, assuming the strong exponential time hypothesis. * New upper and lower bounds for (A)LSH-based algorithms. In particular, we show that asymmetry can be avoided by relaxing the LSH definition to only consider the collision probability of distinct elements. * A new indexing method for IPS based on linear sketches, implying that our hardness results are not far from being tight. Our technical contributions include new asymmetric embeddings that may be of independent interest. At the conceptual level we strive to provide greater clarity, for example by distinguishing among signed and unsigned variants of IPS join and shedding new light on the effect of asymmetry.

The Classic Cross-Correlation and the Real-Valued Jaccard and Coincidence Indices

A Review on the Theory and Application of ICCs

An Analytical Approach to the Jaccard Similarity Index

Partially Proportional and Adaptive Similarity Indices

Multivariate Linear Correlation Analysis

On the Normalization and Visualization of Author Co-Citation Data Salton's Cosine Versus the Jaccard Index

Generalized correlation coefficients of intuitionistic multiplicative sets and their applications to pattern recognition and clustering analysis

Measures of Correlation for Multiple Variables

Partial Distance Correlation with Methods for Dissimilarities

The Jaccard Similarity Mean

Equivalence Of Digital Image Correlation Criteria For Pattern Matching

Comparing high dimensional partitions, with the Coclustering Adjusted Rand Index

The relation between Pearson's correlation coefficient r and Salton's cosine measure

A new correlation coefficient between categorical, ordinal and interval variables with Pearson characteristics

On the Complexity of Inner Product Similarity Join

Measuring Linear Correlation Between Random Vectors

Codivergences and information matrices

Requirements for a cocitation similarity measure, with special reference to Pearson's correlation coefficient

A Spatial Concordance Correlation Coefficient with an Application to Image Analysis

Pearson correlations on complex networks

Compatibility and attainability of matrices of correlation-based measures of concordance