Abstract:This article investigates a relatively underdeveloped subject in natural language processing---the generation of punctuation marks. From a theoretical perspective, we study 16 Chinese punctuation marks as defined in the Chinese national standard of punctuation usage, and categorize these punctuation marks into three different types according to their syntactic properties. We implement a three-tier maximum entropy model incorporating linguistically-motivated features for generating the commonly used Chinese punctuation marks in unpunctuated sentences output by a surface realizer. Furthermore, we present a method to automatically extract cue words indicating sentence-final punctuation marks as a specialized feature to construct a more precise model. Evaluating on the Penn Chinese Treebank data, the MaxEnt model achieves an f -score of 79.83% for punctuation insertion and 74.61% for punctuation restoration using gold data input, 79.50% for insertion and 73.32% for restoration using parser-based imperfect input. The experiments show that the MaxEnt model significantly outperforms a baseline 5-gram language model that scores 54.99% for punctuation insertion and 52.01% for restoration. We show that our results are not far from human performance on the same task with human insertion f -scores in the range of 81-87% and human restoration in the range of 71-82%. Finally, a manual error analysis of the generation output shows that close to 40% of the mismatched punctuation marks do in fact result in acceptable choices, a fact obscured in the automatic string-matching based evaluation scores.

CRF-based Approach to Sentence Segmentation and Punctuation for Ancient Chinese Prose

Method of sentence segmentation and punctuating for ancient Chinese literatures based on cascaded CRF

Predicting Punctuation in Ancient Chinese Texts: A Multi-Layered LSTM and Attention-Based Approach

A Sentence Segmentation Method for Ancient Chinese Texts Based on NNLM.

Ancient Chinese Sentence Segmentation Based on Bidirectional LSTM+CRF Model

Automatic sentence segmentation for classical Chinese: The Spring and Autumn Annals as an example

Classical Chinese Sentence Segmentation for Tomb Biographies of Tang Dynasty

A Chinese Part-of-speech Tagging Approach Using Conditional Random Fields

Pos Tagging For Classical Chinese Text

Chinese Text Chunking Based CRF

Punctuation as implicit annotations for chinese word segmentation

A Linguistically Inspired Statistical Model for Chinese Punctuation Generation

Improving Chinese Word Segmentation on Micro-blog Using Rich Punctuations.

A Sentence Segmentation Method for Ancient Chinese Texts Based on Recurrent Neural Network

The Uncertainty-based Retrieval Framework for Ancient Chinese CWS and POS

Chunk Parsing and Entity Relation Extracting to Chinese Text by Using Conditional Random Fields Model

A morphology-based Chinese word segmentation method

CRFs Based Chinese Word Segmentation

A combination method of CRF with syntactic rules to identify opinion_holder

A Hybrid Sentence Splitting Method By Comma Insertion For Machine Translation With Crf

A Hybrid Approach to Chinese Word Segmentation around CRFs