Abstract:This article presents a pragmatic approach to Chinese word segmentation. It differs from most previous approaches mainly in three respects. First, while theoretical linguists have defined Chinese words using various linguistic criteria, Chinese words in this study are defined pragmatically as segmentation units whose definition depends on how they are used and processed in realistic computer applications. Second, we propose a pragmatic mathematical framework in which segmenting known words and detecting unknown words of different types (i.e., morphologically derived words, factoids, named entities, and other unlisted words) can be performed simultaneously in a unified way. These tasks are usually conducted separately in other systems. Finally, we do not assume the existence of a universal word segmentation standard that is application-independent. Instead, we argue for the necessity of multiple segmentation standards due to the pragmatic fact that different natural language processing applications might require different granularities of Chinese words. These pragmatic approaches have been implemented in an adaptive Chinese word segmenter, called MSRSeg, which will be described in detail. It consists of two components: (1) a generic segmenter that is based on the framework of linear mixture models and provides a unified approach to the five fundamental features of word-level Chinese language processing: lexicon word processing, morphological analysis, factoid detection, named entity recognition, and new word identification; and (2) a set of output adaptors for adapting the output of (1) to different application-specific standards. Evaluation on five test sets with different standards shows that the adaptive system achieves state-of-the-art performance on all the test sets.

A Pragmatic Chinese Word Segmentation System.

Chinese Word Segmentation Based on Mixing Model.

CRF-based Hybrid Model for Word Segmentation, NER and Even POS Tagging

An Improved Chinese Word Segmentation System with Conditional Random Field

Chinese Word Segmentation with Maximum Entropy and N-gram Language Model

Chinese Word Segmentation and Named Entity Recognition: A Pragmatic Approach

Chinese Word Segmentation and Named Entity Recognition Based on a Context-Dependent Mutual Information Independence Model.

Integrating ngram model and case-based learning for Chinese word segmentation

A Study of Chinese Lexical Analysis Based on Discriminative Models

Micro Blogs Oriented Word Segmentation System.

A Hybrid Approach to Chinese Word Segmentation around CRFs

A Pragmatic Approach to Increase Accuracy of Chinese Word-Segmentation

A Pragmatic Approach for Classical Chinese Word Segmentation.

Chinese Word Segmentation with Multiple Postprocessors in HIT-IRLab

A realistic and robust model for Chinese word segmentation

An Improved CRF Based Chinese Language Processing System for SIGHAN Bakeoff 2007.

Chinese Word Segmentation: A Decade Review

Word Segmentation on Micro-blog Texts with External Lexicon and Heterogeneous Data.

The CIPS-SIGHAN CLP 2012 ChineseWord Segmentation Onmicroblog Corpora Bakeoff.

The CIPS-SIGHAN CLP 2014 Chinese Word Segmentation Bake-off.

Chinese word segmentation at Peking University