Abstract:A realistic Chinese word segmentation tool must adapt to textual variations with minimal training input and yet robust enough to yield reliable segmentation result for all variants. Various lexicon-driven approaches to Chinese segmentation, e.g. [1,16], achieve high f-scores yet require massive training for any variation. Text-driven approach, e.g. [12], can be easily adapted for domain and genre changes yet has difficulty matching the high f-scores of the lexicon-driven approaches. In this paper, we refine and implement an innovative text-driven word boundary decision (WBD) segmentation model proposed in [15]. The WBD model treats word segmentation simply and efficiently as a binary decision on whether to realize the natural textual break between two adjacent characters as a word boundary. The WBD model allows simple and quick training data preparation converting characters as contextual vectors for learning the word boundary decision. Machine learning experiments with four different classifiers show that training with 1,000 vectors and 1 million vectors achieve comparable and reliable results. In addition, when applied to SigHAN Bakeoff 3 competition data, the WBD model produces OOV recall rates that are higher than all published results. Unlike all previous work, our OOV recall rate is comparable to our own F-score. Both experiments support the claim that the WBD model is a realistic model for Chinese word segmentation as it can be easily adapted for new variants with the robust result. In conclusion, we will discuss linguistic ramifications as well as future implications for the WBD approach.

A morphology-based Chinese word segmentation method

A Novel Chinese Word Segmentation Method Utilizing Morphology Information

Parsing-based Chinese word segmentation integrating morphological and syntactic information

Chinese word segmentation as morpheme-based lexical chunking

CRFs Based Chinese Word Segmentation

A BiLSTM-CRF Based Approach to Word Segmentation in Chinese

Chinese Tweets Segmentation Based on Morphemes.

Dictionary Chinese Word Segmentation Research a Method Combined with CRFs

Combine CRF and MMSEG to Boost Chinese Word Segmentation in Social Media

CRFs-based Chinese word segmentation method with character position probability feature

A Comparative Study on Chinese Word Segmentation Using Statistical Models

Title Word Structure Analysis Based on Conditional Random Fields

Chinese Word Segmentation Method on the Basis of Bidirectional Long-Short Term Memory Model

Chinese Word Segmentation Via BiLSTM+Semi-CRF with Relay Node

A realistic and robust model for Chinese word segmentation

A Hybrid Approach to Chinese Word Segmentation around CRFs

An Improved Chinese Word Segmentation System with Conditional Random Field

Chinese Word Segmentation based on Word boundary Classificatioin

Joint n-gram Chinese language modeling with an application to Chinese word segmentation

Uyghur Word Segmentation Using a Combination of Rules and Statistics

Chinese Word Segmentation Without Using Lexicon and Hand-Crafted Training Data