Abstract:Word segmentation is an essential and challenging task in natural language processing, especially for the Chinese language due to its high linguistic complexity. Existing methods for Chinese word segmentation, including statistical machine learning methods and neural network methods, usually have good performance in specific knowledge domains. Given the increasing importance of interdisciplinary and cross-domain studies, one of the challenges in cross-domain word segmentation is to handle the out-of-vocabulary (OOV) words. Existing methods show unsatisfactory performance to meet the practical standard. To this end, we propose a document-level context-aware model that can automatically perceive and identify OOV words from different domains. Our method jointly implements a word-based and a character-based model and then processes the results with a newly proposed reconstruction model. We evaluate the new method by designing and conducting comprehensive experiments on two real-world datasets (e.g., news from different domains). The results demonstrate the superiority of our method over the state-of-the-art models in handling texts from different domains. Importantly, when doing the word segmentation under the cross-domain scenario, our proposed method can improve the performance of OOV words recognition.

Stop word list construction and application in Chinese language processing

Automatic Construction of Chinese Stop Word List

Automatic Identification of Stop Words in Chinese Text Classification

New Word Identification in Social Network Text Based on Time Series Information

Chinese Word Segmentation Method Based on Dictionary and Frequency of the Words

Towards Unified Chinese Segmentation Algorithm

Survey on Chinese Word Segmentation

Automatic summarization oriented Chinese word extraction and statistics system

Internet-oriented Chinese New Words Detection

A Design and Test of Chinese Word Segmentation Based on Lucene

Topic Detection Technology for Chinese Text Based on Statistics and Semantic Information

Identification and Extraction of Topic Elements for Chinese Interactive Text

A Survey of Chinese Word Segmentation and the Application in Information Retrieval

News-oriented Automatic Chinese Keyword Indexing

Study of Word-Based Chinese Document Experimental System and Chinese Free-Text Information Extraction Experiment Based on It

Stop Words for Processing Software Engineering Documents: Do they Matter?

Chinese Keyword Extraction Based on N-Gram and Word Co-occurrence

An Automatic Chinese Collocation Extraction Algorithm Based on Lexical Statistics

Method of Special Domain Lexicon Construction Based on Raw Materials

Domain-Aware Word Segmentation for Chinese Language: A Document-Level Context-Aware Model

Query Based Chinese Phrase Extraction for Site Search