Abstract:The lack of word delimiters such as spaces in Chinese texts makes word segmentation a special issue in Chinese text processing. As the volume of Chinese texts grows rapidly on the Internet, the number of unknown words increases accordingly. However, word segmentation approaches relying solely on existing dictionaries are helpless in handling unknown words. In this paper, we propose a novel unsupervised method to segment large Chinese corpora using contextual information. In particular, the number of characters preceding and following a string, known as the accessors of the string, is used to measure the independence of the string. The greater the independence, the more likely it is that the string is a word. The segmentation problem is then considered an optimization problem to maximize the target function of this number over all word candidates in an utterance. Our purpose here is to explore the best function in terms of segmentation performance. The performance is evaluated with the word token recall measure in addition to word type precision and word type recall. Among the three types of target functions that we have explored, polynomial functions turn out to outperform others. This simple method is effective in unsupervised segmentation of Chinese texts and its performance is highly comparable to other recently reported unsupervised segmentation methods.

Algorithm for Solving 3-Character Crossing Ambiguities in Chinese Word Segmentation

Chinese Word Segmentation Method Based on Dictionary and Frequency of the Words

Chinese Word Segmentation Without Using Lexicon and Hand-Crafted Training Data

Towards Unified Chinese Segmentation Algorithm

Research of Reverse Backtracking Matching Algorithm for Chinese Word Segmentation

Chinese Word Segmentation with Character Abstraction.

Resolution to Combinational Ambiguity of Chinese Word Segmentation

Statistical Properties of Overlapping Ambiguities in Chinese Word Segmentation and a Strategy for Their Disambiguation

A Compression-based Algorithm for Chinese Word Segmentation

Some Problems of Chinese Segmentation

Chinese Word Segmentation without Using Dictionary Based on Unsupervised Learning Strategy

A Chinese word segmentation system design based on forward-backward maximum matching algorithm

Segmentation of Connected Chinese Characters Based on Genetic Algorithm

Unsupervised segmentation of chinese corpus using accessor variety

The Research of Chinese Word Segmentation Algorithm Based on Forward Maximum Match

Word Chain-based Automatic Word Segmentation Method

Chinese word segmentation at Peking University

Statistical Properties of Overlapping Word Segmentation Ambiguities in Domain-specific Chinese Corpora

A Statistical Approach For Resolving Problematical Word Boundaries In Chinese Lexicography

Survey on Chinese Word Segmentation

A Unicode Based Adaptive Segmentor