Abstract:The overlapping ambiguity strings(OAS) is one of the difficulties in automatic Chinese word segmentation.This paper treats the resolution of OAS asa classification task,using maximum entropy integrating character features to solve the problem.In order to overcome the data sparseness in maximum entropy modeling,this paper introduces the inequality smoothing techniques and Gaussian smoothing techniques.We compared the Gaussian smoothing,inequality smoothing and frequency discount on the four datasets of the Second International Chinese Word Segmentation,proving that Gaussian smoothing,inequality smoothing are much better than the discount method..while inequality smoothing enables the seamless integration of feature selectioninto the parameter estimation with the result of a significantly compressed model.On the four datasets,the precision of disambiguation by the proposed method can achieve 96.27%,96.83%,96.56%,96.52% respectively,with a relative improvement of 5.87%,5.64%,5.00%,5.00% by the rich feature and a relative improvement of 5.87%,5.64%,5.00%,5.00% by smoothing technology.Meanwhile,the classification models are compressed by 38.7,19.9,44.6,9.7 by using inequality smoothing.

A Practical Approach To Resolving Combination Ambiguity In Chinese Word Segmentation

Resolution to Combinational Ambiguity of Chinese Word Segmentation

Chinese Word Segmentation Method Based on Dictionary and Frequency of the Words

Ambiguity Identification Strategy of Chinese Word Segmentation

Statistical Properties of Overlapping Ambiguities in Chinese Word Segmentation and a Strategy for Their Disambiguation

Resolution to Chinese Combinational Ambiguity Combined Corpus-Based Method with Linguistics Knowledge

A Disabiguation Method for Segmenting Chinese Words by Using Relaxation Algorithm Based on Context

A Resolution Of Overlapping Ambiguity In Tibetan Word Segmentation

Corpus-based method for differentiating genuine and spurious combinational ambiguity

Algorithm for Solving 3-Character Crossing Ambiguities in Chinese Word Segmentation

A search-based Chinese word segmentation method.

A Pragmatic Approach to Increase Accuracy of Chinese Word-Segmentation

Covering ambiguity resolution in Chinese word segmentation based on contextual information

Statistical Properties of Overlapping Word Segmentation Ambiguities in Domain-specific Chinese Corpora

Research of Reverse Backtracking Matching Algorithm for Chinese Word Segmentation

Resolution of Overlapping Ambiguity Strings Based on Smoothed Maximum Entropy Model with Character Feature

A Hybrid Approach to the Real World Text Segmentation

Statistical Approach Based on POS for Chinese Time Word Disambiguity

Solution Strategies for Word Sense Problems Based on Vector Space Model and Maximum Entropy Model

Research on the Application of a Chinese Semantic Knowledge Base in Chinese Phrase Disambiguation

A Comparative Study Of The Effect Of Word Segmentation On Chinese Terminology Extraction