Abstract:Word-level information is important in natural language processing (NLP), especially for the Chinese language due to its high linguistic complexity. Chinese word segmentation (CWS) is an essential task for Chinese downstream NLP tasks. Existing methods have already achieved a competitive performance for CWS on large-scale annotated corpora. However, the accuracy of the method will drop dramatically when it handles an unsegmented text with lots of out-of-vocabulary (OOV) words. In addition, there are many different segmentation criteria for addressing different requirements of downstream NLP tasks. Excessive amounts of models with saving different criteria will generate the explosive growth of the total parameters. To this end, we propose a joint multiple criteria model that shares all parameters to integrate different segmentation criteria into one model. Besides, we utilize a transfer learning method to improve the performance of OOV words. Our proposed method is evaluated by designing comprehensive experiments on multiple benchmark datasets (e.g., Bakeoff 2005, Bakeoff 2008 and SIGHAN 2010). Our method achieves the state-of-the-art performances on all datasets. Importantly, our method also shows a competitive practicability and generalization ability for the CWS task.

Chinese Word Segmentation Method for Domain-Special Machine Translation

Chinese Word Segmentation Method Based on Dictionary and Frequency of the Words

A Chinese Word Segmentation for Statistical Machine Translation

Adapting Chinese Word Segmentation for Translation by Using a Bi- lingual Dictionary

Statistical-based Approach to Chinese Word Segmentation

An empirical study on word segmentation for chinese machine translation

Combining Statistical Model and Dictionary for Domain Adaption of Chinese Word Segmentation

Chinese Word Segmentation Without Using Lexicon and Hand-Crafted Training Data

A Pragmatic Approach to Increase Accuracy of Chinese Word-Segmentation

Chinese word segmenter based on discriminative classifiers integration

A Joint Multiple Criteria Model in Transfer Learning for Cross-domain Chinese Word Segmentation

Towards Accurate Word Segmentation for Chinese Patents

Combining Multiple Chinese Word Segmentation Results for Statistical Machine Translation

The Research of Chinese Automatic Word Segmentation System Based on N-Gram Statistical Model

Improving Cross-Domain Chinese Word Segmentation with Word Embeddings

Probabilistic Chinese word segmentation with non-local information and stochastic training

A Hybrid Approach to the Real World Text Segmentation

Domain Adaptation for CRF-based Chinese Word Segmentation using Free Annotations.

Domain Adaptation for Chinese Word Segmentation Based on Neural Network

Increasing Accuracy of Chinese Segmentation with Strategy of Multi-step Processing

Adaptive Multi-Task Transfer Learning for Chinese Word Segmentation in Medical Text.