Abstract:In natural language processing (NLP), text similarity calculation is widely used in information retrieval, machine translation, text mining etc. The definition of the similarity between texts may not just refer to words with similar meanings. Domain similarity, which evaluates the similarity on basis of domain reference, is becoming a promising approach in dealing with large documents. By adopting domain similarity calculation, the degree of similarity could be controlled at different semantic levels, and extract texts in different domain granularity. For example, web pages of Lakers, NBA, basketball and sports could be retrieved respectively with different settings of domain similarity. LSI (Latent Semantic Indexing) is a feasible approach that can be applied to calculate text domain similarity. By controlling the number of topics, the domain similarity can be determined in different granularity. However, the performance is greatly affected by the number of specified topics, which is required for LSI algorithm. In this paper, an adaptive method was applied to word similarity calculation. TF-IDF was used to get the word frequency in the text, and the number of topics in the mixed text, set by dimensionality reduction and clustering was automatically obtained. According to the number of clusters, the similarity between text domains was calculated as the number of topics mapped to the subspace in the LSI. Accordingly, experimental results have shown that the method proposed in this paper is superior to other algorithms in the accuracy of text similarity calculation.

Automatic Domain-Specific Term Extraction and Its Application in Text Classification

Automatic Extraction of Domain-Specific Terms

An adaptive method for text domain similarity calculation

Automatic term extraction from Chinese scientific texts

Extracting Domain-Specific Terms From Unlabeled Web Documents By Bootstrapping And Term Classifiers

A Novel Topic Model for Automatic Term Extraction

Research on Deep Web Classification Based on Domain Feature Text

Improving Short Text Classification Through Better Feature Space Selection

Text Categorization Based on Domain Ontology

Research On Automatic Acquisition Of Domain Terms

Research on Automatic Chinese Multi-word Term Extraction Based on Integration of Web Information and Term Component

A Survey of Term Recognition and Extraction for Domainspecific Chinese Text Information Processing

Research on Automatic Chinese Multi-word Term Extraction Based on Term Component

Measuring Termhood in Automatic Terminology Extraction

Parsing-based Automatic Chinese Term Extraction

Research on Domain Term Extraction Based on Conditional Random Fields

Topic Correlation Analysis for Cross-Domain Text Classification

Domain-Specific Sentiment Word Extraction by Seed Expansion and Pattern Generation

On the Unsupervised Analysis of Domain-Specific Chinese Texts

Chinese Terminology Extraction Using EM-Based Transfer Learning Method.

Improved Automatic Keyword Extraction Based on TextRank Using Domain Knowledge