Abstract:A realistic Chinese word segmentation tool must adapt to textual variations with minimal training input and yet robust enough to yield reliable segmentation result for all variants. Various lexicon-driven approaches to Chinese segmentation, e.g. [1,16], achieve high f-scores yet require massive training for any variation. Text-driven approach, e.g. [12], can be easily adapted for domain and genre changes yet has difficulty matching the high f-scores of the lexicon-driven approaches. In this paper, we refine and implement an innovative text-driven word boundary decision (WBD) segmentation model proposed in [15]. The WBD model treats word segmentation simply and efficiently as a binary decision on whether to realize the natural textual break between two adjacent characters as a word boundary. The WBD model allows simple and quick training data preparation converting characters as contextual vectors for learning the word boundary decision. Machine learning experiments with four different classifiers show that training with 1,000 vectors and 1 million vectors achieve comparable and reliable results. In addition, when applied to SigHAN Bakeoff 3 competition data, the WBD model produces OOV recall rates that are higher than all published results. Unlike all previous work, our OOV recall rate is comparable to our own F-score. Both experiments support the claim that the WBD model is a realistic model for Chinese word segmentation as it can be easily adapted for new variants with the robust result. In conclusion, we will discuss linguistic ramifications as well as future implications for the WBD approach.

Feature-rich Prepositional Phrase Boundary Identification based on SVM

ON DETECTION OF PROSODIC PHRASE BOUNDARIES IN SPONTANEOUS SPEECH

A SVM-Based Model for Chinese Functional Chunk Parsing.

Implementing Chinese new word discovery and POS tagging based on support vector machine

Investigating Effect of Rich Syntactic Features on Mandarin Prosodic Boundaries Prediction

Chinese event descriptive clause splitting with structured SVMs

Detecting Syntactic Features of Translated Chinese

An Algorithm for Automatic Recognition of Chinese Spatial Prepositional Phrases for Academic Texts

Chinese WSD based on features obtaining with shallow parsing

Query Based Chinese Phrase Extraction for Site Search

A realistic and robust model for Chinese word segmentation

Chinese word segmentation at Peking University

Automatic detection of contrastive word pairs using textual and acoustic features

Prosodic Word Boundaries Prediction for Mandarin Text-to-Speech

Automatic Recognition of Verb-Complement Separable Words Based on BCC

Acoustic and Linguistic Information Based Chinese Prosodic Boundary Labelling

Chinese Prosodic Phrasing with Extended Features.

CFN-based Semantic Role Labeling of Chinese Prepositional Phrase ⋆

Multi-Class Part Parsing with Joint Boundary-Semantic Awareness.

Unsupervised Prosodic Phrase Boundary Labeling Of Mandarin Speech Synthesis Database Using Context-Dependent Hmm

A Statistical Approach For Resolving Problematical Word Boundaries In Chinese Lexicography