Abstract:Word segmentation is an essential and challenging task in natural language processing, especially for the Chinese language due to its high linguistic complexity. Existing methods for Chinese word segmentation, including statistical machine learning methods and neural network methods, usually have good performance in specific knowledge domains. Given the increasing importance of interdisciplinary and cross-domain studies, one of the challenges in cross-domain word segmentation is to handle the out-of-vocabulary (OOV) words. Existing methods show unsatisfactory performance to meet the practical standard. To this end, we propose a document-level context-aware model that can automatically perceive and identify OOV words from different domains. Our method jointly implements a word-based and a character-based model and then processes the results with a newly proposed reconstruction model. We evaluate the new method by designing and conducting comprehensive experiments on two real-world datasets (e.g., news from different domains). The results demonstrate the superiority of our method over the state-of-the-art models in handling texts from different domains. Importantly, when doing the word segmentation under the cross-domain scenario, our proposed method can improve the performance of OOV words recognition.

Covering ambiguity resolution in Chinese word segmentation based on contextual information

Context-based Approach to Covering Ambiguity Resolution in Chinese Word Segmentation

Resolution to Combinational Ambiguity of Chinese Word Segmentation

Chinese Word Segmentation Method Based on Dictionary and Frequency of the Words

Statistical Properties of Overlapping Ambiguities in Chinese Word Segmentation and a Strategy for Their Disambiguation

A Resolution Of Overlapping Ambiguity In Tibetan Word Segmentation

Statistical Properties of Overlapping Word Segmentation Ambiguities in Domain-specific Chinese Corpora

Ambiguity Identification Strategy of Chinese Word Segmentation

Resolution to Chinese Combinational Ambiguity Combined Corpus-Based Method with Linguistics Knowledge

Parsing-based Chinese word segmentation integrating morphological and syntactic information

Algorithm for Solving 3-Character Crossing Ambiguities in Chinese Word Segmentation

A Discriminative Latent Variable Chinese Segmenter with Hybrid Word/Character Information.

Research and Implementation of Related Algorithm of Chinese Text Categorization

A lexicalized second-order-HMM for ambiguity resolution in Chinese segmentation and POS tagging

Chinese Word Sense Disambiguation Based on Context Expansion.

A Segmentation Matrix Method for Chinese Segmentation Ambiguity Analysis.

Unsupervised Word Sense Disambiguation Using Neighborhood Knowledge.

Chinese Word Sense Disambiguation Based on Extension Theory

A Novel Approach to Word Sense Disambiguation Based on Topical and Semantic Association

Domain-Aware Word Segmentation for Chinese Language: A Document-Level Context-Aware Model

A Disabiguation Method for Segmenting Chinese Words by Using Relaxation Algorithm Based on Context