Abstract:Document similarity search is to find documents similar to a given query document and return a ranked list of similar documents to users, which is widely used in many text and web systems, such as digital library, search engine, etc. Traditional retrieval models, including the Okapi's BM25 model and the Smart's vector space model with length normalization, could handle this problem to some extent by taking the query document as a long query. In practice, the Cosine measure is considered as the best model for document similarity search because of its good ability to measure similarity between two documents. In this paper, the quantitative performances of the above models are compared using experiments. Because the Cosine measure is not able to reflect the structural similarity between documents, a new retrieval model based on TextTiling is proposed in the paper. The proposed model takes into account the subtopic structures of documents. It first splits the documents into text segments with TextTiling and calculates the similarities for different pairs of text segments in the documents. Lastly the overall similarity between the documents is returned by combining the similarities of different pairs of text segments with optimal matching method. Experiments are performed and results show: 1) the popular retrieval models (the Okapi's BM25 model and the Smart's vector space model with length normalization) do not perform well for document similarity search; 2) the proposed model based on TextTiling is effective and outperforms other models, including the Cosine measure; 3) the methods for the three components in the proposed model are validated to be appropriately employed.

Texts retrieval algorithm combined with texts importance

Automatic Retrieval Method for Tracing Links Between Code and Chinese Documentation

Using Anchor Text Refined by Page Importance to Improve Web Retrieval

A Novel Full-Text Indexing Model for Chinese Text Retrieval

Chinese Text Retrieval: an Adjacency Matrix Based Approach

Improving Web Image Retrieval Using Image Annotations and Inference Network

Using the Dempster-Shafer Theory of Evidence to Rank Documents

RankTopic: Ranking Based Topic Modeling

Improvement of Text Similarity Computing Algorithm Based on Attribute

A New Retrieval Model Based on TextTiling for Document Similarity Search

A New Full-Text Indexing Model with Low Space Overhead for Chinese Text Retrieval

Clustering web images by correlation mining of image-text

NERank: Bringing Order to Named Entities from Texts.

Text Retrieval Technology Based on Keyword Retrieval

Extract Salient Words with WordRank for Effective Similarity Search in Text Data

PostingRank: Bringing Order to Web Forum Postings

A Term Co-Occurrence Algorithm and the Effect of Co-Occurrence Terms on Result Ranking for Information Retrieval

The enhancement of TextRank algorithm by using word2vec and its application on topic extraction

Improving web search results using affinity graph.

Text clustering based on the user search intention

The Effect of the Multi-Layer Text Summarization Model on the Efficiency and Relevancy of the Vector Space-based Information Retrieval