Abstract:The language model design and implementation issue is researched in this paper. Different from previous research, we want to emphasize the importance of n-gram models based on words in the study of language model. We build up a word based language model using the toolkit of SRILM and implement it for contextual language processing on Chinese documents. A modified Absolute Discount smoothing algorithm is proposed to reduce the perplexity of the language model. The word based language model improves the performance of post-processing of online handwritten character recognition system compared with the character based language model, but it also increases computation and storage cost greatly. Besides quantizing the model data non-uniformly, we design a new tree storage structure to compress the model size, which leads to an increase in searching efficiency as well. We illustrate the set of approaches on a test corpus of recognition results of online handwritten Chinese characters, and propose a modified confidence measure for recognition candidate characters to get their accurate posterior probabilities while reducing the complexity. The weighted combination of linguistic knowledge and candidate confidence information proves successful in this paper and can be further developed to achieve improvements in recognition accuracy.

Chinese Organization Name Automatic Recognition Using Class-based Language Model

A Simple Rule-Based Approach To Organization Name Recognition In Chinese Text

Automatic Recognition of Chinese Organization Name Based on Cascaded Conditional Random Fields

Study on Approaches to Retrieval of Chinese Organization Name Based on Its Abbreviated Name

A Two-Stage Bi-LSTM Model for Chinese Company Name Recognition.

Chinese Named Entity Recognition with a Multi-Phase Model

Automatic Chinese name recognition based on web corpus analysis

SAN: Structure-Aware Network for Complex and Long-tailed Chinese Text Recognition

A Multilevel Approach For Recognizing Characters In Engineering Drawings

Chinese Named Entity Recognition and Disambiguation Based on Multi-stage Clustering

An Efficient Post-Processing Approach for Off-Line Handwritten Chinese Address Recognition

A Word Language Model Based Contextual Language Processing On Chinese Character Recognition

Study on Hierarchical Speech Recognition

Chinese Clinical Named Entity Recognition with ALBERT and MHA Mechanism

A Post-processing Approach for Handwritten Chinese Address Recognition

Me-Based Chinese Person Name and Location Name Recognition Model

A Toponym Entity Recognition Model for Chinese Social Media

A Chinese named entity recognition model incorporating recurrent cell and information state recursion

Integrating Boundary Assembling into a DNN Framework for Named Entity Recognition in Chinese Social Media Text

Multi-document Chinese Name Disambiguation Based on Latent Semantic Analysis

ELCA: Enhanced boundary location for Chinese named entity recognition via contextual association