Abstract:The overlapping ambiguity strings(OAS) is one of the difficulties in automatic Chinese word segmentation.This paper treats the resolution of OAS asa classification task,using maximum entropy integrating character features to solve the problem.In order to overcome the data sparseness in maximum entropy modeling,this paper introduces the inequality smoothing techniques and Gaussian smoothing techniques.We compared the Gaussian smoothing,inequality smoothing and frequency discount on the four datasets of the Second International Chinese Word Segmentation,proving that Gaussian smoothing,inequality smoothing are much better than the discount method..while inequality smoothing enables the seamless integration of feature selectioninto the parameter estimation with the result of a significantly compressed model.On the four datasets,the precision of disambiguation by the proposed method can achieve 96.27%,96.83%,96.56%,96.52% respectively,with a relative improvement of 5.87%,5.64%,5.00%,5.00% by the rich feature and a relative improvement of 5.87%,5.64%,5.00%,5.00% by smoothing technology.Meanwhile,the classification models are compressed by 38.7,19.9,44.6,9.7 by using inequality smoothing.

Statistical Approach Based on POS for Chinese Time Word Disambiguity

Chinese Word Segmentation Method Based on Dictionary and Frequency of the Words

Statistical Properties of Overlapping Ambiguities in Chinese Word Segmentation and a Strategy for Their Disambiguation

Statistical Properties of Overlapping Word Segmentation Ambiguities in Domain-specific Chinese Corpora

Statistical-based Approach to Chinese Word Segmentation

An Enhanced Model for Chinese Word Segmentation and Part-of-Speech Tagging.

Resolution of Overlapping Ambiguity Strings Based on Smoothed Maximum Entropy Model with Character Feature

Ambiguity Identification Strategy of Chinese Word Segmentation

Resolution to Combinational Ambiguity of Chinese Word Segmentation

Unknown Word Processing In Hmm-Based Pos Tagging

Research on the Approach of Integrating Chinese Word Segmentation with Part-of-speech Tagging

The Research of Chinese Automatic Word Segmentation System Based on N-Gram Statistical Model

Increasing Accuracy of Chinese Segmentation with Strategy of Multi-step Processing

Joint Chinese Word Segmentation and POS Tagging System with Undirected Graphical Models

Research of Chinese Word Segmentation Algorithms Based on Statistics and Reverse Maximum Match

A Comparison Study of Sequence Labeling Methods for Chinese Word Segmentation,POS Tagging Models

Exploring Multiple Features for POS Guessing of Chinese Unknown Words with Maximum Entropy Models

A Comparative Study on Chinese Word Segmentation Using Statistical Models

Improving Chinese POS Tagging with Dependency Parsing.

A Statistical Approach For Resolving Problematical Word Boundaries In Chinese Lexicography

Chinese syntactic category disambiguation using support vector machines