Abstract:Not all languages, e.g. Chinese, have delimiters for words. To extract words from a sentence in these languages, we usually rely on a dictionary for known words. For unknown words, some approaches rely on a domain specific dictionary or a tailor-made learning data set. However, this information may not be available. Another direction is to use unsupervised methods. These methods rely on a goodness measure to evaluate how likely the words are meaningful based on a statistical argument on the given text. The most challenging issue is to identify low-frequency meaningful words. In this paper, we first show by an empirical study on Chinese texts that all classical goodness measures cannot separate low-frequency meaningful and meaningless words effectively. To solve this problem, we propose a new goodness measure, the overlap variety method. The key idea behind the new measure is not to consider the absolute number of occurrences of the candidate (i.e., a string of Chinese characters) but to compare the goodness measures (we use the accessor variety) of the candidate and those of the strings overlapping the candidate. The candidate is likely to be meaningful if its accessor variety is larger than the accessor varieties of the overlapping strings. We implement an extraction system for unknown Chinese word, UNExtract, based on this overlap variety method. We evaluate our approach using the CIPS-SIGHAN-2010 bake off corpora and show that the proposed measure is more effective than the other five state-of-the-art goodness measures (accessor variety, branch entropy, description length gain, frequency substring reduction, pointwise mutual information), especially for low-frequency words and bi-gram words.

Exploring Multiple Features for Sense Prediction of Chinese Unknown Words.

Exploring Multiple Features for POS Guessing of Chinese Unknown Words with Maximum Entropy Models

Do Multi-Sense Embeddings Improve Natural Language Understanding?

Fusion of Multiple Features and Supervised Learning for Chinese OOV Term Detection and POS Guessing.

Coarse-Grained Word Sense Disambiguation Using Features Described in the Lexicon

Unknown Chinese Word Extraction Based on Variety of Overlapping Strings

Learning Sense-specific Word Embeddings By Exploiting Bilingual Resources.

Solution Strategies for Word Sense Problems Based on Vector Space Model and Maximum Entropy Model

Combining Contextual And Structural Information For Supersense Tagging Of Chinese Unknown Words

Multi-strategy approach to Chinese word sense disambiguation based on sememe relations

Unsupervised segmentation of chinese corpus using accessor variety

A Study in Dictionary-Based All-word Word Sense Disambiguation for Pre-Qin Chinese

Chinese Word Sense Disambiguation Based on Extension Theory

Leveraging Human Prior Knowledge to Learn Sense Representations

On Modeling Sense Relatedness in Multi-prototype Word Embedding.

A Unified Model for Word Sense Representation and Disambiguation.

Using a Chinese Lexicon to Learn Sense Embeddings and Measure Semantic Similarity.

Leveraging Word-Formation Knowledge for Chinese Word Sense Disambiguation

Application of the transformer model algorithm in chinese word sense disambiguation: a case study in chinese language

A method of Part-Of-Speech guessing of Chinese Unknown Words based on combined features

Chinese Word Sense Disambiguation Using a LSTM