Abstract:A realistic Chinese word segmentation tool must adapt to textual variations with minimal training input and yet robust enough to yield reliable segmentation result for all variants. Various lexicon-driven approaches to Chinese segmentation, e.g. [1,16], achieve high f-scores yet require massive training for any variation. Text-driven approach, e.g. [12], can be easily adapted for domain and genre changes yet has difficulty matching the high f-scores of the lexicon-driven approaches. In this paper, we refine and implement an innovative text-driven word boundary decision (WBD) segmentation model proposed in [15]. The WBD model treats word segmentation simply and efficiently as a binary decision on whether to realize the natural textual break between two adjacent characters as a word boundary. The WBD model allows simple and quick training data preparation converting characters as contextual vectors for learning the word boundary decision. Machine learning experiments with four different classifiers show that training with 1,000 vectors and 1 million vectors achieve comparable and reliable results. In addition, when applied to SigHAN Bakeoff 3 competition data, the WBD model produces OOV recall rates that are higher than all published results. Unlike all previous work, our OOV recall rate is comparable to our own F-score. Both experiments support the claim that the WBD model is a realistic model for Chinese word segmentation as it can be easily adapted for new variants with the robust result. In conclusion, we will discuss linguistic ramifications as well as future implications for the WBD approach.

Incorporate Web Search Technology to Solve Out-of-Vocabulary Words in Chinese Word Segmentation.

A Unified Model for Solving the OOV Problem of Chinese Word Segmentation

Domain-Aware Word Segmentation for Chinese Language: A Document-Level Context-Aware Model

Automatic Corpus Expansion for Chinese Word Segmentation by Exploiting the Redundancy of Web Information.

Neural Chinese Word Segmentation with Dictionary Knowledge

Chinese Word Segmentation: Another Decade Review (2007-2017)

Towards Unified Chinese Segmentation Algorithm

Chinese Word Segmentation with Heterogeneous Graph Neural Network

Enhancing Chinese Word Segmentation Using Unlabeled Data

A New Error-driven Learning Approach for Chinese Word Segmentation

Improving Cross-Domain Chinese Word Segmentation with Word Embeddings

Feature Abstraction for Lightweight and Accurate Chinese Word Segmentation

Is Word Segmentation Necessary for Deep Learning of Chinese Representations?

Neural Chinese Word Segmentation as Sequence to Sequence Translation

RethinkCWS: is Chinese Word Segmentation a Solved Task?

Chinese Word Segmentation Via BiLSTM+Semi-CRF with Relay Node

Neural Networks Incorporating Unlabeled and Partially-labeled Data for Cross-domain Chinese Word Segmentation

A realistic and robust model for Chinese word segmentation

Unsupervised Chinese Word Segmentation with BERT Oriented Probing and Transformation

Robust Chinese Word Segmentation with Contextualized Word Representations

Word Segmentation for Chinese Novels