Abstract:The lack of word delimiters such as spaces in Chinese texts makes word segmentation a special issue in Chinese text processing. As the volume of Chinese texts grows rapidly on the Internet, the number of unknown words increases accordingly. However, word segmentation approaches relying solely on existing dictionaries are helpless in handling unknown words. In this paper, we propose a novel unsupervised method to segment large Chinese corpora using contextual information. In particular, the number of characters preceding and following a string, known as the accessors of the string, is used to measure the independence of the string. The greater the independence, the more likely it is that the string is a word. The segmentation problem is then considered an optimization problem to maximize the target function of this number over all word candidates in an utterance. Our purpose here is to explore the best function in terms of segmentation performance. The performance is evaluated with the word token recall measure in addition to word type precision and word type recall. Among the three types of target functions that we have explored, polynomial functions turn out to outperform others. This simple method is effective in unsupervised segmentation of Chinese texts and its performance is highly comparable to other recently reported unsupervised segmentation methods.

Method of Recognizing Unknown Words by Building Single-Word Dictionary

Chinese Word Segmentation Method Based on Dictionary and Frequency of the Words

Chinese Word Segmentation Probability Dictionary Training and Enrich Solution

Neural Networks Incorporating Dictionaries for Chinese Word Segmentation.

New Word Identification in Social Network Text Based on Time Series Information

Neural Chinese Word Segmentation with Dictionary Knowledge

Chinese Word Segmentation without Using Dictionary Based on Unsupervised Learning Strategy

Chinese Word Segmentation Without Using Lexicon and Hand-Crafted Training Data

Towards Unified Chinese Segmentation Algorithm

On-line Handwritten Chinese Word Recognition Based on Lexicon

TopWORDS-Seg: Simultaneous Text Segmentation and Word Discovery for Open-Domain Chinese Texts via Bayesian Inference

Chinese word segmentation as morpheme-based lexical chunking

Unsupervised segmentation of chinese corpus using accessor variety

Chinese Word Segmentation with Character Abstraction.

Research on Chinese Automatic Segmentation

Research on Intelligent Construction of China English Network New Words Database Based on Adjacent Entropy Recognition Algorithm

A New Error-driven Learning Approach for Chinese Word Segmentation

Incorporate Web Search Technology to Solve Out-of-Vocabulary Words in Chinese Word Segmentation.

A Method Combining Rule-based and Statistics-based Approaches for Chinese Word Segmentation

Chinese word segmentation at Peking University

A Large-Vocabulary Chinese Speech Recognition System.