Abstract:Chinese new words and their part-of-speech (POS) are particularly problematic in Chinese natural language processing. With the fast development of internet and information technology, it is impossible to get a complete system dictionary for Chinese natural language processing, as new words out of the basic system dictionary are always being created. A latent semi-CRF model, which combines the strengths of LDCRF (Latent-Dynamic Conditional Random Field) and semi-CRF, is proposed to detect the new words together with their POS synchronously regardless of the types of the new words from the Chinese text without being pre-segmented. Unlike the original semi-CRF, the LDCRF is applied to generate the candidate entities for training and testing the latent semi-CRF, which accelerates the training speed and decreases the computation cost. The complexity of the latent semi-CRF could be further adjusted by tuning the number of hidden variables in LDCRF and the number of the candidate entities from the Nbest outputs of the LDCRF. A new-words-generating framework is proposed for model training and testing, under which the definitions and distributions of the new words conform to the ones existing in real text. Specific features called “Global Fragment Information” for new word detection and POS tagging are adopted in the model training and testing. The experimental results show that the proposed method is capable of detecting even low frequency new words together with their POS tags. The proposed model is found to be performing competitively with the state-of-the-art models presented.

HMM and CRF Based Hybrid Model for Chinese Lexical Analysis

Chinese Lexical Analysis Based On Hybrid Mmsm Model

Research on Chinese Lexical Analysis System by Fusing Multiple Knowledge Sources

A Hybrid Approach to Chinese Word Segmentation around CRFs

A Study of Chinese Lexical Analysis Based on Discriminative Models

CRF-based Hybrid Model for Word Segmentation, NER and Even POS Tagging

An Improved CRF Based Chinese Language Processing System for SIGHAN Bakeoff 2007.

Chinese lexical analysis based on hidden semi-crf

LM Enhanced BiRNN-CRF for Joint Chinese Word Segmentation and POS Tagging.

Chinese Named Entity Recognition and Word Segmentation Based on Character.

Joint n-gram Chinese language modeling with an application to Chinese word segmentation

Chinese Word Segmentation with Maximum Entropy and N-gram Language Model

Detecting New Words from Chinese Text Using Latent Semi-CRF Models.

Bidirectional LSTM-CRF Attention-based Model for Chinese Word Segmentation

A Word Language Model Based Contextual Language Processing On Chinese Character Recognition

A BiLSTM-CRF Based Approach to Word Segmentation in Chinese

A Discriminative Latent Variable Chinese Segmenter with Hybrid Word/Character Information.

Leveraging Rich Linguistic Features for Cross-domain Chinese Segmentation.

A Hybrid Post-Processing System For Offline Handwritten Chinese Character Recognition Based On A Statistical Language Model

Chinese Named Entity Recognition with a Multi-Phase Model

Chinese Text Chunking Using Lexicalized HMMs