Abstract:A set of IR experiments was carried out to study the impact of Chinese word segmentation and its effect on information retrieval (IR) at the Division of Information Studies, Nanyang Technological University, Singapore. A total of four automatic character-based segmentation approaches and a manual word segmentation approach was first carried out to obtain the word segments for indexing and to evaluate the segmentation accuracy of these automatic approaches. The IR experiments study both the influence of different document segmentation approaches on IR effectiveness and the methods used for query segmentation. Traditional data recall and precision measures were used to gauge IR effectiveness. A number of queries were selected and subjected to further detailed analysis to further explore the influence of word segmentation on IR.The findings reveal that the segmentation approach has an effect on IR effectiveness. Better IR results are obtained by using the same method for query and document processing as this increase the probability of the query-document match. The recognition of a higher number of 2-character words generally contributes to the improvement of IR effectiveness. However, manual segmentation does not always work better than character-based segmentation as a result of the existence of longer words with more than two characters. No evidence is found that ambiguous words resulting from the segmentation process significantly affect IR.

A Survey of Chinese Word Segmentation and the Application in Information Retrieval

Chinese word segmentation and its effect on information retrieval

Chinese Word Segmentation Method Based on Dictionary and Frequency of the Words

Survey on Chinese Word Segmentation

Research on Chinese Automatic Segmentation

The Locations of Word Segmentation in Chinese Reading: Research Based on the Eye-Movement-Contingent Display Technique

Chinese word segmentation at Peking University

Chinese Word Segmentation Evaluation Methodology Based on Web Search Engines

Towards Unified Chinese Segmentation Algorithm

A Hybrid Approach to Chinese Word Segmentation around CRFs

Automatic summarization oriented Chinese word extraction and statistics system

Some Problems of Chinese Segmentation

Unsupervised segmentation of chinese corpus using accessor variety

A Method Combining Rule-based and Statistics-based Approaches for Chinese Word Segmentation

Chinese Word Segmentation with Character Abstraction.

Automatic Corpus Expansion for Chinese Word Segmentation by Exploiting the Redundancy of Web Information.

Chinese Word Segmentation Without Using Lexicon and Hand-Crafted Training Data

A Pragmatic Approach for Classical Chinese Word Segmentation.

Research on Automatic Recognition of Separable Words in Modern Chinese

Researches on Full Text Retrieval

Segmentation of Chinese Discourse in Content-Based Information Retrieval.