Abstract:The lack of word delimiters such as spaces in Chinese texts makes word segmentation a special issue in Chinese text processing. As the volume of Chinese texts grows rapidly on the Internet, the number of unknown words increases accordingly. However, word segmentation approaches relying solely on existing dictionaries are helpless in handling unknown words. In this paper, we propose a novel unsupervised method to segment large Chinese corpora using contextual information. In particular, the number of characters preceding and following a string, known as the accessors of the string, is used to measure the independence of the string. The greater the independence, the more likely it is that the string is a word. The segmentation problem is then considered an optimization problem to maximize the target function of this number over all word candidates in an utterance. Our purpose here is to explore the best function in terms of segmentation performance. The performance is evaluated with the word token recall measure in addition to word type precision and word type recall. Among the three types of target functions that we have explored, polynomial functions turn out to outperform others. This simple method is effective in unsupervised segmentation of Chinese texts and its performance is highly comparable to other recently reported unsupervised segmentation methods.

Extract Chinese Unknown Words from a Large-scale Corpus Using Morphological and Distributional Evidences.

Automatic Corpus Expansion for Chinese Word Segmentation by Exploiting the Redundancy of Web Information.

Unsupervised segmentation of chinese corpus using accessor variety

A Unified Model for Solving the OOV Problem of Chinese Word Segmentation

Incorporate Web Search Technology to Solve Out-of-Vocabulary Words in Chinese Word Segmentation.

A morphology-based Chinese word segmentation method

Towards Unified Chinese Segmentation Algorithm

Chinese word segmentation as morpheme-based lexical chunking

A WSD Model for Corpus Construction

Feature Abstraction for Lightweight and Accurate Chinese Word Segmentation

A Local Generative Model For Chinese Word Segmentation

Neural Chinese Word Segmentation with Dictionary Knowledge

A Joint Model for Unsupervised Chinese Word Segmentation.

Parsing-based Chinese word segmentation integrating morphological and syntactic information

Multiple Character Embeddings for Chinese Word Segmentation

Detecting new Chinese words from massive domain texts with word embedding

Improving Cross-Domain Chinese Word Segmentation with Word Embeddings

Enhancing Chinese Word Segmentation Using Unlabeled Data

Accessor variety criteria for Chinese word extraction

Word frequency approximation for chinese using raw, MM-Segmented and manually segmented corpora

Chinese Word Frequency Approximation Based on Multitype Corpora.