Abstract:Document similarity search is to find documents similar to a given query document and return a ranked list of similar documents to users, which is widely used in many text and web systems, such as digital library, search engine, etc. Traditional retrieval models, including the Okapi's BM25 model and the Smart's vector space model with length normalization, could handle this problem to some extent by taking the query document as a long query. In practice, the Cosine measure is considered as the best model for document similarity search because of its good ability to measure similarity between two documents. In this paper, the quantitative performances of the above models are compared using experiments. Because the Cosine measure is not able to reflect the structural similarity between documents, a new retrieval model based on TextTiling is proposed in the paper. The proposed model takes into account the subtopic structures of documents. It first splits the documents into text segments with TextTiling and calculates the similarities for different pairs of text segments in the documents. Lastly the overall similarity between the documents is returned by combining the similarities of different pairs of text segments with optimal matching method. Experiments are performed and results show: 1) the popular retrieval models (the Okapi's BM25 model and the Smart's vector space model with length normalization) do not perform well for document similarity search; 2) the proposed model based on TextTiling is effective and outperforms other models, including the Cosine measure; 3) the methods for the three components in the proposed model are validated to be appropriately employed.

Beyond Tf-Idf And Cosine Distance In Documents Dissimilarity Measure

A New Simple and Effective Measure for Bag-of-word Inter-Document Similarity Measurement.

An adaptive method for text domain similarity calculation

M(P)-Dissimilarity: A Data Dependent Dissimilarity Measure

A measure based on optimal matching in graph theory for document similarity

Exploiting Category Information and Document Information to Improve Term Weighting for Text Categorization

TF-IDFC-RF: A Novel Supervised Term Weighting Scheme

A Comparative Study of Data-Dependent Approaches Without Learning in Measuring Similarities of Data Objects.

Data-dependent Dissimilarity Measure: an Effective Alternative to Geometric Distance Measures

Semantic smoothing of document models for agglomerative clustering

Short text similarity measurement using context‐aware weighted biterms

Beyond topical similarity: a structural similarity measure for retrieving highly similar documents

Several alternative term weighting methods for text representation and classification

A New Retrieval Model Based on TextTiling for Document Similarity Search

A novel document similarity measure based on earth mover's distance

Wasserstein-Fisher-Rao Document Distance

The earth mover's distance as a semantic measure for document similarity.

A Systematic Comparison of Data Selection Criteria for SMT Domain Adaptation

Modified DFS-based term weighting scheme for text classification

Research on Text Similarity Measurement Hybrid Algorithm with Term Semantic Information and TF-IDF Method

A Novel Term Weighting Scheme for Automated Text Categorization