Abstract:A realistic Chinese word segmentation tool must adapt to textual variations with minimal training input and yet robust enough to yield reliable segmentation result for all variants. Various lexicon-driven approaches to Chinese segmentation, e.g. [1,16], achieve high f-scores yet require massive training for any variation. Text-driven approach, e.g. [12], can be easily adapted for domain and genre changes yet has difficulty matching the high f-scores of the lexicon-driven approaches. In this paper, we refine and implement an innovative text-driven word boundary decision (WBD) segmentation model proposed in [15]. The WBD model treats word segmentation simply and efficiently as a binary decision on whether to realize the natural textual break between two adjacent characters as a word boundary. The WBD model allows simple and quick training data preparation converting characters as contextual vectors for learning the word boundary decision. Machine learning experiments with four different classifiers show that training with 1,000 vectors and 1 million vectors achieve comparable and reliable results. In addition, when applied to SigHAN Bakeoff 3 competition data, the WBD model produces OOV recall rates that are higher than all published results. Unlike all previous work, our OOV recall rate is comparable to our own F-score. Both experiments support the claim that the WBD model is a realistic model for Chinese word segmentation as it can be easily adapted for new variants with the robust result. In conclusion, we will discuss linguistic ramifications as well as future implications for the WBD approach.

Improved Viterbi Algorithm-Based HMM2 for Chinese Words Segmentation

Chinese Word Segmentation Method Based on Dictionary and Frequency of the Words

A Hierarchical Viterbi Algorithm For Mandarin Hybrid Speech Synthesis System

A Discriminative Latent Variable Chinese Segmenter with Hybrid Word/Character Information.

Automatic speech segmentation for Chinese speech database based on HMM

VB-HMM Speaker Diarization with Enhanced and Refined Segment Representation.

Chinese Vowels Recognition Method Based on Non-Homogeneous Hidden Markov Model

A Refined Hdp-Based Model For Unsupervised Chinese Word Segmentation

A Comparative Study on Chinese Word Segmentation Using Statistical Models

Discriminative Training and Explicit Duration Modeling for HMM-based Automatic Segmentation

Human-Computer Interactive Chinese Word Segmentation: an Adaptive Dirichlet Process Mixture Model Approach.

A Novel Short Merged Off-line Handwritten Chinese Character String Segmentation Algorithm Using Hidden Markov Model

Segmentation and Tagging Ambiguity for Chinese Using HMM

Chinese Word Segmentation with Maximum Entropy and N-gram Language Model

A Joint Model for Unsupervised Chinese Word Segmentation.

Improvement of hidden Markov model (HMM) for speech recognition

A realistic and robust model for Chinese word segmentation

Word frequency approximation for chinese using raw, MM-Segmented and manually segmented corpora

Automatic Phonetic Segmentation Using HMM Model

Enhancing Domain Portability of Chinese Segmentation Model Using Chi-Square Statistics and Bootstrapping

A hybrid Chinese word segmentation model for quality management-related texts based on transfer learning