Abstract:A realistic Chinese word segmentation tool must adapt to textual variations with minimal training input and yet robust enough to yield reliable segmentation result for all variants. Various lexicon-driven approaches to Chinese segmentation, e.g. [1,16], achieve high f-scores yet require massive training for any variation. Text-driven approach, e.g. [12], can be easily adapted for domain and genre changes yet has difficulty matching the high f-scores of the lexicon-driven approaches. In this paper, we refine and implement an innovative text-driven word boundary decision (WBD) segmentation model proposed in [15]. The WBD model treats word segmentation simply and efficiently as a binary decision on whether to realize the natural textual break between two adjacent characters as a word boundary. The WBD model allows simple and quick training data preparation converting characters as contextual vectors for learning the word boundary decision. Machine learning experiments with four different classifiers show that training with 1,000 vectors and 1 million vectors achieve comparable and reliable results. In addition, when applied to SigHAN Bakeoff 3 competition data, the WBD model produces OOV recall rates that are higher than all published results. Unlike all previous work, our OOV recall rate is comparable to our own F-score. Both experiments support the claim that the WBD model is a realistic model for Chinese word segmentation as it can be easily adapted for new variants with the robust result. In conclusion, we will discuss linguistic ramifications as well as future implications for the WBD approach.

Chinese Word Segmentation of Ideological and Political Education Based on Unsupervised Learning

The Application of Unsupervised Learning TF-IDF Algorithm in Word Segmentation of Ideological and Political Education

Chinese Word Segmentation Method Based on Dictionary and Frequency of the Words

Research on the Influence of Socialist Core Value System Construction on College Students’ Ideological and Political Education in the Context of Deep Learning

Word Segmentation for Classical Chinese Buddhist Literature

Unsupervised Word Segmentation with Bi-directional Neural Language Model

Unsupervised Learning helps Supervised Neural Word Segmentation

Unsupervised segmentation of chinese corpus using accessor variety

Chinese Word Segmentation without Using Dictionary Based on Unsupervised Learning Strategy

Chinese word segmentation at Peking University

Neural Word Segmentation Learning for Chinese

Chinese Word Segmentation Probability Dictionary Training and Enrich Solution

Chinese Word Segmentation with Maximum Entropy and N-gram Language Model

Word Segmentation as Graph Partition.

A realistic and robust model for Chinese word segmentation

Long Short-Term Memory Neural Networks for Chinese Word Segmentation.

Exploiting Unlabeled Text with Different Unsupervised Segmentation Criteria for Chinese Word Segmentation

Unsupervised Chinese Word Segmentation with BERT Oriented Probing and Transformation

A Local Generative Model For Chinese Word Segmentation

How unsupervised learning affects character tagging based Chinese Word Segmentation: A quantitative investigation

Unsupervised Neural Word Segmentation for Chinese Via Segmental Language Modeling