Abstract:The language model design and implementation issue is researched in this paper. Different from previous research, we want to emphasize the importance of n-gram models based on words in the study of language model. We build up a word based language model using the toolkit of SRILM and implement it for contextual language processing on Chinese documents. A modified Absolute Discount smoothing algorithm is proposed to reduce the perplexity of the language model. The word based language model improves the performance of post-processing of online handwritten character recognition system compared with the character based language model, but it also increases computation and storage cost greatly. Besides quantizing the model data non-uniformly, we design a new tree storage structure to compress the model size, which leads to an increase in searching efficiency as well. We illustrate the set of approaches on a test corpus of recognition results of online handwritten Chinese characters, and propose a modified confidence measure for recognition candidate characters to get their accurate posterior probabilities while reducing the complexity. The weighted combination of linguistic knowledge and candidate confidence information proves successful in this paper and can be further developed to achieve improvements in recognition accuracy.

Semantic model for Chinese phoneme-to-character transcription

A High Accuracy Approach for Word-Phoneme Translation Using Neural Networks

Two Language Models Using Chinese Semantic Parsing

A new linguistic decoding method for online handwritten Chinese character recognition

Integrated Semantic and Phonetic Post-correction for Chinese Speech Recognition

New Perspectives in Sinographic Language Processing Through the Use of Character Structure

Joint-Character-Poc N-Gram Language Modeling for Chinese Speech Recognition

Word extraction based on semantic constraints in chinese word-formation

Phonology and access to Chinese character meaning.

SEMI-CLASS-BASED N-GRAM LANGUAGE MODELING FOR CHINESE DICTATION

Exploiting Word Semantics to Enrich Character Representations of Chinese Pre-trained Models

Semantic-Aware Chinese Zero Pronoun Resolution with Pre-trained Semantic Dependency Parser.

A Hybrid Semantic and Word Based Language Model and Its Applications

A Chinese OCR Spelling Check Approach Based on Statistical Language Models.

Parsing-based Chinese word segmentation integrating morphological and syntactic information

Optimal Model for Handwritten Chinese Character Segmention

What Is the Basic Semantic Unit of Chinese Language? A Computational Approach Based on Topic Models.

A New Word Language Model Evaluation Metric For Character Based Languages

A Word Language Model Based Contextual Language Processing On Chinese Character Recognition

Character-Level Syntax Infusion in Pre-Trained Models for Chinese Semantic Role Labeling.

MANDARIN PRONUNCIATION VARIATION MODELING 1