Abstract:Not all languages, e.g. Chinese, have delimiters for words. To extract words from a sentence in these languages, we usually rely on a dictionary for known words. For unknown words, some approaches rely on a domain specific dictionary or a tailor-made learning data set. However, this information may not be available. Another direction is to use unsupervised methods. These methods rely on a goodness measure to evaluate how likely the words are meaningful based on a statistical argument on the given text. The most challenging issue is to identify low-frequency meaningful words. In this paper, we first show by an empirical study on Chinese texts that all classical goodness measures cannot separate low-frequency meaningful and meaningless words effectively. To solve this problem, we propose a new goodness measure, the overlap variety method. The key idea behind the new measure is not to consider the absolute number of occurrences of the candidate (i.e., a string of Chinese characters) but to compare the goodness measures (we use the accessor variety) of the candidate and those of the strings overlapping the candidate. The candidate is likely to be meaningful if its accessor variety is larger than the accessor varieties of the overlapping strings. We implement an extraction system for unknown Chinese word, UNExtract, based on this overlap variety method. We evaluate our approach using the CIPS-SIGHAN-2010 bake off corpora and show that the proposed measure is more effective than the other five state-of-the-art goodness measures (accessor variety, branch entropy, description length gain, frequency substring reduction, pointwise mutual information), especially for low-frequency words and bi-gram words.

Exploring Multiple Features for POS Guessing of Chinese Unknown Words with Maximum Entropy Models

A method of Part-Of-Speech guessing of Chinese Unknown Words based on combined features

Applying Multi-features on POS Tag Guessing of Unknown Chinese Words

Chinese POS Tagging Based on Maximum Entropy Model

Fusion of Multiple Features and Supervised Learning for Chinese OOV Term Detection and POS Guessing.

Chinese Pos Tagging Using Restricted Maximum Entropy Model

An Enhanced Model for Chinese Word Segmentation and Part-of-Speech Tagging.

Unknown Word Processing In Hmm-Based Pos Tagging

Applying Class Triggers in Chinese Pos Tagging Based on Maximum Entropy Model

Unknown Chinese Word Extraction Based on Variety of Overlapping Strings

Chinese Word Segmentation Via Word-position Tagging Based on Maximum Entropy Model

A Two-Stage Approach to Chinese Part-of-Speech Tagging

Improving Chinese POS Tagging with Dependency Parsing.

A lexicalized second-order-HMM for ambiguity resolution in Chinese segmentation and POS tagging

Automatic Recognition of Chinese Unknown Word for Single-Character and Affix Models

Chinese Word Segmentation with Maximum Entropy and N-gram Language Model

A probabilistic feature based maximum entropy model for chinese named entity recognition

Parsing-based Chinese word segmentation integrating morphological and syntactic information

Experimental Study of Hidden Markov Model Based Part-of-speech Tagging for Chinese Texts

Part-of-speech Tagger Based on Maximum Entropy Model

Chinese Lexical Analysis Using Hierarchical Hidden Markov Model