Abstract:In natural language processing (NLP), text similarity calculation is widely used in information retrieval, machine translation, text mining etc. The definition of the similarity between texts may not just refer to words with similar meanings. Domain similarity, which evaluates the similarity on basis of domain reference, is becoming a promising approach in dealing with large documents. By adopting domain similarity calculation, the degree of similarity could be controlled at different semantic levels, and extract texts in different domain granularity. For example, web pages of Lakers, NBA, basketball and sports could be retrieved respectively with different settings of domain similarity. LSI (Latent Semantic Indexing) is a feasible approach that can be applied to calculate text domain similarity. By controlling the number of topics, the domain similarity can be determined in different granularity. However, the performance is greatly affected by the number of specified topics, which is required for LSI algorithm. In this paper, an adaptive method was applied to word similarity calculation. TF-IDF was used to get the word frequency in the text, and the number of topics in the mixed text, set by dimensionality reduction and clustering was automatically obtained. According to the number of clusters, the similarity between text domains was calculated as the number of topics mapped to the subspace in the LSI. Accordingly, experimental results have shown that the method proposed in this paper is superior to other algorithms in the accuracy of text similarity calculation.

Text Segmentation with LDA-based Fisher Kernel

CLDA: Feature Selection for Text Categorization Based on Constrained LDA

Comprehending international important Ramsar wetland documents using latent semantic topic model in kernel space: LIN et al.

An adaptive method for text domain similarity calculation

Integrating Data-Driven Segmentation, Local Feature Extraction and Fisher Kernel Encoding to Improve Time Series Classification

Latent Fisher Discriminant Analysis

Using multiple discriminant analysis approach for linear text segmentation

πLDA: document clustering with selective structural constraints.

Interactive Image Segmentation Based on Samples Reconstruction and FLDA

A Hybrid Approach to the Real World Text Segmentation

Face Recognition Using Novel LDA-Based Algorithms.

A unified framework for interactive image segmentation via Fisher rules

A Discriminative Latent Variable Chinese Segmenter with Hybrid Word/Character Information.

Web Text Classification based on LDA Model

Intelligent Segmentation Framework and Data Hierarchy of Chinese Language and Literature Based on Semantic Recognition

An automatic approach for efficient text segmentation

Study on influences of different Chinese word segmentation methods to text automatic classification based on LDA model

Human-Computer Interactive Chinese Word Segmentation: an Adaptive Dirichlet Process Mixture Model Approach.

Enhancing Lstm-Based Word Segmentation Using Unlabeled Data

Probabilistic Chinese word segmentation with non-local information and stochastic training

Using Topic Labels for Text Summarization.