Abstract:As Chinese is an ideographic character-based language, the words in the texts are not delimited by spaces. Indexing of Chinese documents is impossible without a proper segmentation algorithm. Many Chinese segmentation algorithms have been proposed in the past. Traditional segmentation algorithms cannot operate without a large dictionary or a large corpus of training data. Nowadays, the Web has become the largest corpus that is ideal for Chinese segmentation. Although the search engines do not segment texts into proper words, they maintain huge databases of documents and frequencies of character sequences in the documents. Their databases are important potential resources for segmentation. In this paper, we propose a segmentation algorithm by mining web data with the help from search engines. It is the first unified segmentation algorithm for Chinese language from different geographical areas. Experiments have been conducted on the datasets of a recent Chinese segmentation competition. The results show that our algorithm outperforms the traditional algorithms in terms of precision and recall. Moreover, our algorithm can effectively deal with the problem of segmentation ambiguity, new word (unknown word) detection, and stop words.

Some Problems of Chinese Segmentation

Chinese Word Segmentation Method Based on Dictionary and Frequency of the Words

The Locations of Word Segmentation in Chinese Reading: Research Based on the Eye-Movement-Contingent Display Technique

Research on Chinese Automatic Segmentation

Survey on Chinese Word Segmentation

Towards Unified Chinese Segmentation Algorithm

Algorithm for Solving 3-Character Crossing Ambiguities in Chinese Word Segmentation

Chinese word segmentation at Peking University

Chinese Word Segmentation with Character Abstraction.

Resolution to Combinational Ambiguity of Chinese Word Segmentation

A Chinese word segmentation system design based on forward-backward maximum matching algorithm

Effects of prosodic patterns and the morpheme position probability on word segmentation and recognition in overlapping ambiguous strings by learners of Chinese

Chinese Word Segmentation Without Using Lexicon and Hand-Crafted Training Data

A Survey of Chinese Word Segmentation and the Application in Information Retrieval

A Chinese Word Segmentation for Statistical Machine Translation

Blending segmentation with tagging in Chinese language corpus processing

A Statistical Approach For Resolving Problematical Word Boundaries In Chinese Lexicography

Research of Reverse Backtracking Matching Algorithm for Chinese Word Segmentation

Identification of Chinese Personal Names in Unrestricted Texts

Page Segmentation of Chinese Newspapers

A Discriminative Latent Variable Chinese Segmenter with Hybrid Word/Character Information.