Abstract:The lack of word delimiters such as spaces in Chinese texts makes word segmentation a special issue in Chinese text processing. As the volume of Chinese texts grows rapidly on the Internet, the number of unknown words increases accordingly. However, word segmentation approaches relying solely on existing dictionaries are helpless in handling unknown words. In this paper, we propose a novel unsupervised method to segment large Chinese corpora using contextual information. In particular, the number of characters preceding and following a string, known as the accessors of the string, is used to measure the independence of the string. The greater the independence, the more likely it is that the string is a word. The segmentation problem is then considered an optimization problem to maximize the target function of this number over all word candidates in an utterance. Our purpose here is to explore the best function in terms of segmentation performance. The performance is evaluated with the word token recall measure in addition to word type precision and word type recall. Among the three types of target functions that we have explored, polynomial functions turn out to outperform others. This simple method is effective in unsupervised segmentation of Chinese texts and its performance is highly comparable to other recently reported unsupervised segmentation methods.

Unsupervised Learning helps Supervised Neural Word Segmentation

Enhancing Chinese Word Segmentation Using Unlabeled Data

Unsupervised Word Segmentation with Bi-directional Neural Language Model

Unsupervised Neural Word Segmentation for Chinese Via Segmental Language Modeling

Neural Chinese Word Segmentation with Lexicon and Unlabeled Data via Posterior Regularization

Segment, Mask, and Predict: Augmenting Chinese Word Segmentation with Self-Supervision

Neural Networks Incorporating Unlabeled and Partially-labeled Data for Cross-domain Chinese Word Segmentation

Neural Word Segmentation Learning for Chinese

Improving Chinese Word Segmentation Using Partially Annotated Sentences

Long Short-Term Memory Neural Networks for Chinese Word Segmentation.

Unsupervised segmentation of chinese corpus using accessor variety

Neural Chinese Word Segmentation with Dictionary Knowledge

Exploring Representations from Unlabeled Data with Co-training for Chinese Word Segmentation.

Unsupervised Chinese Word Segmentation with BERT Oriented Probing and Transformation

DAG-based Long Short-Term Memory for Neural Word Segmentation

Deep Learning for Chinese Word Segmentation and POS Tagging.

Chinese Word Segmentation without Using Dictionary Based on Unsupervised Learning Strategy

Improving Cross-Domain Chinese Word Segmentation with Word Embeddings

Neural Networks Incorporating Dictionaries for Chinese Word Segmentation.

Semi-Supervised Mesh Segmentation and Labeling

Unsupervised Feature Learning for Chinese Lexicon Based on Auto-Encoder