Abstract:A realistic Chinese word segmentation tool must adapt to textual variations with minimal training input and yet robust enough to yield reliable segmentation result for all variants. Various lexicon-driven approaches to Chinese segmentation, e.g. [1,16], achieve high f-scores yet require massive training for any variation. Text-driven approach, e.g. [12], can be easily adapted for domain and genre changes yet has difficulty matching the high f-scores of the lexicon-driven approaches. In this paper, we refine and implement an innovative text-driven word boundary decision (WBD) segmentation model proposed in [15]. The WBD model treats word segmentation simply and efficiently as a binary decision on whether to realize the natural textual break between two adjacent characters as a word boundary. The WBD model allows simple and quick training data preparation converting characters as contextual vectors for learning the word boundary decision. Machine learning experiments with four different classifiers show that training with 1,000 vectors and 1 million vectors achieve comparable and reliable results. In addition, when applied to SigHAN Bakeoff 3 competition data, the WBD model produces OOV recall rates that are higher than all published results. Unlike all previous work, our OOV recall rate is comparable to our own F-score. Both experiments support the claim that the WBD model is a realistic model for Chinese word segmentation as it can be easily adapted for new variants with the robust result. In conclusion, we will discuss linguistic ramifications as well as future implications for the WBD approach.

Adapting Chinese Word Segmentation for Translation by Using a Bi- lingual Dictionary

Chinese Word Segmentation Method Based on Dictionary and Frequency of the Words

A Chinese Word Segmentation for Statistical Machine Translation

Chinese Word Segmentation Without Using Lexicon and Hand-Crafted Training Data

Neural Networks Incorporating Dictionaries for Chinese Word Segmentation.

Recognition and Segmentation of English Long and Short Sentences Based on Machine Translation

A Unicode Based Adaptive Segmentor

Segmenting Long Sentence Pairs for Statistical Machine Translation

A Discriminative Latent Variable Chinese Segmenter with Hybrid Word/Character Information.

Chinese Word Segmentation Probability Dictionary Training and Enrich Solution

Finding Better Subword Segmentation for Neural Machine Translation

A realistic and robust model for Chinese word segmentation

A Statistical Approach For Resolving Problematical Word Boundaries In Chinese Lexicography

A Method Combining Rule-based and Statistics-based Approaches for Chinese Word Segmentation

Adapting Conventional Chinese Word Segmenter for Segmenting Micro-blog Text: Combining Rule-based and Statistic-based Approaches.

Chinese Word Segmentation without Using Dictionary Based on Unsupervised Learning Strategy

Domain adaptation for statistical machine translation with domain dictionary and monolingual corpora

Probabilistic Chinese word segmentation with non-local information and stochastic training

Chinese word segmentation as morpheme-based lexical chunking

Enhancing Statistical Machine Translation with Character Alignment

Human-Computer Interactive Chinese Word Segmentation: an Adaptive Dirichlet Process Mixture Model Approach.