Abstract:A set of IR experiments was carried out to study the impact of Chinese word segmentation and its effect on information retrieval (IR) at the Division of Information Studies, Nanyang Technological University, Singapore. A total of four automatic character-based segmentation approaches and a manual word segmentation approach was first carried out to obtain the word segments for indexing and to evaluate the segmentation accuracy of these automatic approaches. The IR experiments study both the influence of different document segmentation approaches on IR effectiveness and the methods used for query segmentation. Traditional data recall and precision measures were used to gauge IR effectiveness. A number of queries were selected and subjected to further detailed analysis to further explore the influence of word segmentation on IR.The findings reveal that the segmentation approach has an effect on IR effectiveness. Better IR results are obtained by using the same method for query and document processing as this increase the probability of the query-document match. The recognition of a higher number of 2-character words generally contributes to the improvement of IR effectiveness. However, manual segmentation does not always work better than character-based segmentation as a result of the existence of longer words with more than two characters. No evidence is found that ambiguous words resulting from the segmentation process significantly affect IR.

A Comparative Study Of The Effect Of Word Segmentation On Chinese Terminology Extraction

Chinese Word Segmentation Method Based on Dictionary and Frequency of the Words

An empirical study on word segmentation for chinese machine translation

Survey on Chinese Word Segmentation

Research on Chinese Automatic Segmentation

A search-based Chinese word segmentation method.

Chinese word segmentation and its effect on information retrieval

Chinese Terminology Extraction Using Window-Based Contextual Information

Chinese word segmentation at Peking University

A Method Combining Rule-based and Statistics-based Approaches for Chinese Word Segmentation

A Pragmatic Approach to Increase Accuracy of Chinese Word-Segmentation

A Hybrid Approach to the Real World Text Segmentation

A Survey of Chinese Word Segmentation and the Application in Information Retrieval

Chinese Terminology Extraction Using Bilingual Web Resources

Statistical-based Approach to Chinese Word Segmentation

A Comparative Study on Chinese Word Segmentation Using Statistical Models

Chinese Word Segmentation with Character Abstraction.

Word Segmentation for Chinese Novels

Domain Term Extraction Method Based on Hierarchical Combination Strategy for Chinese Web Documents

A Design and Test of Chinese Word Segmentation Based on Lucene