Adaptive Chinese Word Segmentation with Online Passive-Aggressive Algorithm.

Wenjun Gao,Xipeng Qiu,Xuanjing Huang

2010-01-01

Abstract:In this paper, we describe our system 1 for CIPS-SIGHAN-2010 bake-off task of Chinese word segmentation, which focused on the cross-domain performance of Chinese word segmentation algorithms. We use the online passive-aggressive algorithm with domain invariant information for cross-domain Chinese word segmentation.

What problem does this paper attempt to address?

The CIPS-SIGHAN CLP 2012 ChineseWord Segmentation Onmicroblog Corpora Bakeoff.

Huiming Duan,Zhifang Sui,Ye Tian,Wenjie Li

2012-01-01

Abstract:The CIPS-SIGHAN CLP 2012 Chinese Word Segmentation on MicroBlog Cor- pora Bakeoff was held in the autumn of 2012. This bake-off task of Chinese word segmentation is focused on the perfor- mance of Chinese word segmentation al- gorithms on MicroBlog corpora. 17 groups submitted 20 results, among which the best system has all the P, R and F values near 95%, and the average values of the 17 systems are 0.8931, 0.8981 and 0.8953, respectively.
The CIPS-SIGHAN CLP 2014 Chinese Word Segmentation Bake-off.

Huiming Duan,Zhifang Sui,Tao Ge

DOI: https://doi.org/10.3115/v1/w14-6814

2014-01-01

Abstract:This paper summarizes the SIGHAN 2014 Chinese Word Segmentation bakeoff in several aspects such as dataset, evaluation results.In addition, we analyze errors of segmentation by instance and make a suggestion for improving segmentation systems.
Addressing Domain Adaptation for Chinese Word Segmentation with Global Recurrent Structure.

Shen Huang,Xu Sun,Houfeng Wang

2017-01-01

Abstract:Boundary features are widely used in traditional Chinese Word Segmentation (CWS) methods as they can utilize unlabeled data to help improve the Out-of-Vocabulary (OOV) word recognition performance. Although various neural network methods for CWS have achieved performance competitive with state-of-the-art systems, these methods, constrained by the domain and size of the training corpus, do not work well in domain adaptation. In this paper, we propose a novel BLSTM-based neural network model which incorporates a global recurrent structure designed for modeling boundary features dynamically. Experiments show that the proposed structure can effectively boost the performance of Chinese Word Segmentation, especially OOV-Recall, which brings benefits to domain adaptation. We achieved state-of-the-art results on 6 domains of CNKI articles, and competitive results to the best reported on the 4 domains of SIGHAN Bakeoff 2010 data.
A New Error-driven Learning Approach for Chinese Word Segmentation

XIA Xin-Song,XIAO Jian-Guo

2006-01-01

Computer Science

Abstract:A well known problem for Chinese word segmentation(CWS)is that we can not have a unique definition of words.Different standards may result in different word segmentation outputs.It is unrealizable to develop different CWS systems according to different applications or standards,so it is significantly important to flexibly adapt segmen- tation outputs towards different standards or applications using existing CWS system.The paper presents a linguistical- ly enriched transformation-based learning approach for performing CWS adaptation as a postprocessor.Different from other transform-based learning used in CWS,the approach utilizes some linguistics information,and introduces word class and word internal structure to rule templates and transformations.The performance of the approach is evaluated on four different test sets,which represent four different standards.It turns out to be comparable to several state-of- the-art approaches which perform Chinese word segmentation based on single standard.
A realistic and robust model for Chinese word segmentation

Chu-Ren Huang,Ting-Shuo Yo,Petr Simon,Shu-Kai Hsieh

DOI: https://doi.org/10.48550/arXiv.1905.08732

2019-05-22

Abstract:A realistic Chinese word segmentation tool must adapt to textual variations with minimal training input and yet robust enough to yield reliable segmentation result for all variants. Various lexicon-driven approaches to Chinese segmentation, e.g. [1,16], achieve high f-scores yet require massive training for any variation. Text-driven approach, e.g. [12], can be easily adapted for domain and genre changes yet has difficulty matching the high f-scores of the lexicon-driven approaches. In this paper, we refine and implement an innovative text-driven word boundary decision (WBD) segmentation model proposed in [15]. The WBD model treats word segmentation simply and efficiently as a binary decision on whether to realize the natural textual break between two adjacent characters as a word boundary. The WBD model allows simple and quick training data preparation converting characters as contextual vectors for learning the word boundary decision. Machine learning experiments with four different classifiers show that training with 1,000 vectors and 1 million vectors achieve comparable and reliable results. In addition, when applied to SigHAN Bakeoff 3 competition data, the WBD model produces OOV recall rates that are higher than all published results. Unlike all previous work, our OOV recall rate is comparable to our own F-score. Both experiments support the claim that the WBD model is a realistic model for Chinese word segmentation as it can be easily adapted for new variants with the robust result. In conclusion, we will discuss linguistic ramifications as well as future implications for the WBD approach.

Computation and Language,Information Retrieval,Machine Learning
Adapting Conventional Chinese Word Segmenter for Segmenting Micro-blog Text: Combining Rule-based and Statistic-based Approaches.

Ning Xi,Bin Li,Guangchao Tang,Shujian Huang,Yinggong Zhao,Hao Zhou,Xinyu Dai,Jiajun Chen

2012-01-01

Abstract:We describe two adaptation strategies which are used in our word segmentation system in participating the Microblog word segmentation bake-off: Domain invariant information is extracted from the in-domain unlabelled corpus, and is incorporated as supplementary features to conventional word segmenter based on Conditional Random Field (CRF), we call it statistic-based adaptation. Some heuristic rules are further used to post-process the word segmentation result in order to better handle the characters in emoticons, name entities and special punctuation patterns which extensively exist in micro-blog text, and we call it rule-based adaptation. Experimentally, using both adaptation strategies, our system achieved 92.46 points of F-score, compared with 88.73 points of F-score of the unadapted CRF word segmenter on the pre-released development data. Our system achieved 92.51 points of F-score on the final test data.
Improving Cross-Domain Chinese Word Segmentation with Word Embeddings

Yuxiao Ye,Yue Zhang,Weikang Li,Likun Qiu,Jian Sun

DOI: https://doi.org/10.18653/v1/N19-1279

2019-03-29

Abstract:Cross-domain Chinese Word Segmentation (CWS) remains a challenge despite recent progress in neural-based CWS. The limited amount of annotated data in the target domain has been the key obstacle to a satisfactory performance. In this paper, we propose a semi-supervised word-based approach to improving cross-domain CWS given a baseline segmenter. Particularly, our model only deploys word embeddings trained on raw text in the target domain, discarding complex hand-crafted features and domain-specific dictionaries. Innovative subsampling and negative sampling methods are proposed to derive word embeddings optimized for CWS. We conduct experiments on five datasets in special domains, covering domains in novels, medicine, and patent. Results show that our model can obviously improve cross-domain CWS, especially in the segmentation of domain-specific noun entities. The word F-measure increases by over 3.0% on four datasets, outperforming state-of-the-art semi-supervised and unsupervised cross-domain CWS approaches with a large margin. We make our code and data available on Github.

Computation and Language
Chinese Word Segmentation with Heterogeneous Graph Neural Network

Xuemei Tang,Jun Wang,Qi Su

DOI: https://doi.org/10.48550/arXiv.2201.08975

2022-01-22

Computation and Language

Abstract:In recent years, deep learning has achieved significant success in the Chinese word segmentation (CWS) task. Most of these methods improve the performance of CWS by leveraging external information, e.g., words, sub-words, syntax. However, existing approaches fail to effectively integrate the multi-level linguistic information and also ignore the structural feature of the external information. Therefore, in this paper, we proposed a framework to improve CWS, named HGNSeg. It exploits multi-level external information sufficiently with the pre-trained language model and heterogeneous graph neural network. The experimental results on six benchmark datasets (e.g., Bakeoff 2005, Bakeoff 2008) validate that our approach can effectively improve the performance of Chinese word segmentation. Importantly, in cross-domain scenarios, our method also shows a strong ability to alleviate the out-of-vocabulary (OOV) problem.
Automatic Corpus Expansion for Chinese Word Segmentation by Exploiting the Redundancy of Web Information.

Xipeng Qiu,Chaochao Huang,Xuanjing Huang

2014-01-01

Abstract:Currently most of state-of-the-art methods for Chinese word segmentation (CWS) are based on supervised learning, which depend on large scale annotated corpus. However, these supervised methods do not work well when we deal with a new different domain without enough annotated corpus. In this paper, we propose a method to automatically expand the training corpus for the out-of-domain texts by exploiting the redundant information on Web. We break up a complex and uncertain segmentation by resorting to Web for an ample supply of relevant easy-to-segment sentences. Then we can pick out some reliable segmented sentences and add them to corpus. With the augmented corpus, we can re-train a better segmenter to resolve the original complex segmentation. The experimental results show that our approach can more effectively and stably improve the performance of CWS. Our method also provides a new viewpoint to enhance the performance of CWS by automatically expanding corpus rather than developing complicated algorithms or features.
Probabilistic Chinese word segmentation with non-local information and stochastic training

Xu Sun,Yaozhong Zhang,Takuya Matsuzaki,Yoshimasa Tsuruoka,Jun’ichi Tsujii

DOI: https://doi.org/10.1016/j.ipm.2012.12.003

IF: 7.466

2013-01-01

Information Processing & Management

Abstract:In this article, we focus on Chinese word segmentation by systematically incorporating non-local information based on latent variables and word-level features. Differing from previous work which captures non-local information by using semi-Markov models, we propose an alternative method for modeling non-local information: a latent variable word segmenter employing word-level features. In order to reduce computational complexity of learning non-local information, we further present an improved online training method, which can arrive the same objective optimum with a significantly accelerated training speed. We find that the proposed method can help the learning of long range dependencies and improve the segmentation quality of long words (for example, complicated named entities). Experimental results demonstrate that the proposed method is effective. With this improvement, evaluations on the data of the second SIGHAN CWS bakeoff show that our system is competitive with the state-of-the-art systems.
Feature Abstraction for Lightweight and Accurate Chinese Word Segmentation

Le Tian,Xipeng Qiu,Xuanjing Huang

DOI: https://doi.org/10.1109/ialp.2013.65

2013-01-01

Abstract:Chinese word segmentation (CWS) is an important and necessary problem to analyze Chinese texts. The state-of-art CWS systems are mostly based on sequence labeling algorithm and use the discriminative model with millions of overlapping binary features. However, there are few works on porting these systems to the devices with limited computing capacity and memory. In this paper, we focus on two challenges in Chinese word segmentation: (1) low accuracy of out-of-vocabulary word and (2) huge feature space. To resolve these two difficult problems, we propose a method to abstract the original input on both character and feature levels. We group the ``similar'' features to generate more abstract representation. Experimental results show that feature abstraction can greatly reduce the feature space with a comparable performance.
An Improved CRF Based Chinese Language Processing System for SIGHAN Bakeoff 2007.

Xihong Wu,Xiaojun Lin,Xinhao Wang,Chunyao Wu,Yaozhong Zhang,Dianhai Yu

2008-01-01

Abstract:This paper describes three systems: the Chinese word segmentation (WS) system, the named entity recognition (NER) system and the Part-of-Speech tagging (POS) system, which are submitted to the Fourth International Chinese Language Processing Bakeoff. Here, Conditional Random Fields (CRFs) are employed as the primary models. For the WS and NER tracks, the ngram language model is incorporated in our CRFs based systems in order to take into account the higher level language information. Furthermore, to improve the performances of our submitted systems, a transformationbased learning (TBL) technique is adopted for post-processing.
A Hybrid Approach to Chinese Word Segmentation around CRFs

ZHOU Jun-sheng,DAI Xin-yu,NI Rui-yu,CHEN Jia-jun

2005-01-01

Abstract:In this paper, we present a Chinese word segmentation system which is consisted of four components, i.e. basic segmentation, named entity recognition, error-driven learner and new word detector. The basic segmentation and named entity recognition, implemented based on conditional random fields, are used to generate initial segmentation results. The other two components are used to refine the results. Our system participated in the tests on open and closed tracks of Beijing University (PKU) and Microsoft Research (MSR). The actual evaluation results show that our system performs very well in MSR open track, MSR closed track and PKU open track.
Toward Fast and Accurate Neural Chinese Word Segmentation with Multi-Criteria Learning

Weipeng Huang,Xingyi Cheng,Kunlong Chen,Taifeng Wang,Wei Chu

DOI: https://doi.org/10.48550/arXiv.1903.04190

2019-03-11

Computation and Language

Abstract:The ambiguous annotation criteria lead to divergence of Chinese Word Segmentation (CWS) datasets in various granularities. Multi-criteria Chinese word segmentation aims to capture various annotation criteria among datasets and leverage their common underlying knowledge. In this paper, we propose a domain adaptive segmenter to exploit diverse criteria of various datasets. Our model is based on Bidirectional Encoder Representations from Transformers (BERT), which is responsible for introducing open-domain knowledge. Private and shared projection layers are proposed to capture domain-specific knowledge and common knowledge, respectively. We also optimize computational efficiency via distillation, quantization, and compiler optimization. Experiments show that our segmenter outperforms the previous state of the art (SOTA) models on 10 CWS datasets with superior efficiency.
A Long Dependency Aware Deep Architecture for Joint Chinese Word Segmentation and POS Tagging.

Xinchi Chen,Xipeng Qiu,Xuanjing Huang

2016-01-01

Abstract:Long-term context is crucial to joint Chinese word segmentation and POS tagging (Su0026T) task. However, most of machine learning based methods extract features from a window of characters. Due to the limitation of window size, these methods can not exploit the long distance information. In this work, we propose a long dependency aware deep architecture for joint Su0026T task. Specifically, to simulate the feature templates of traditional discrete feature based models, we use different filters to model the complex compositional features with convolutional and pooling layer, and then utilize long distance dependency information with recurrent layer. Experiment results on five different datasets show the effectiveness of our proposed model.
Adversarial Multi-Criteria Learning for Chinese Word Segmentation

Xinchi Chen,Zhan Shi,Xipeng Qiu,Xuanjing Huang

DOI: https://doi.org/10.48550/arXiv.1704.07556

2017-04-25

Computation and Language

Abstract:Different linguistic perspectives causes many diverse segmentation criteria for Chinese word segmentation (CWS). Most existing methods focus on improve the performance for each single criterion. However, it is interesting to exploit these different criteria and mining their common underlying knowledge. In this paper, we propose adversarial multi-criteria learning for CWS by integrating shared knowledge from multiple heterogeneous segmentation criteria. Experiments on eight corpora with heterogeneous segmentation criteria show that the performance of each corpus obtains a significant improvement, compared to single-criterion learning. Source codes of this paper are available on Github.
Chinese Word Segmentation Without Using Lexicon and Hand-Crafted Training Data

Maosong Sun,Dayang Shen,Benjamin K. Tsou

DOI: https://doi.org/10.3115/980691.980775

1998-01-01

Abstract:Chinese word segmentation is the first step in any Chinese NLP system. This paper presents a new algorithm for segmenting Chinese texts without making use of any lexicon and hand-crafted linguistic resource. The statistical data required by the algorithm, that is, mutual information and the difference of t-score between characters, is derived automatically from raw Chinese corpora. The preliminary experiment shows that the segmentation accuracy of our algorithm is acceptable. We hope the gaining of this approach will be beneficial to improving the performance (especially in ability to cope with unknown words and ability to adapt to various domains) of the existing segmenters, though the algorithm itself can also be utilized as a stand-alone segmenter in some NLP applications.
A Discriminative Latent Variable Chinese Segmenter with Hybrid Word/Character Information.

Xu Sun,Yao-zhong Zhang,Takuya Matsuzaki,Yoshimasa Tsuruoka,Jun'ichi Tsujii

DOI: https://doi.org/10.3115/1620754.1620763

2009-01-01

Abstract:Conventional approaches to Chinese word segmentation treat the problem as a character-based tagging task. Recently, semi-Markov models have been applied to the problem, incorporating features based on complete words. In this paper, we propose an alternative, a latent variable model, which uses hybrid information based on both word sequences and character sequences. We argue that the use of latent variables can help capture long range dependencies and improve the recall on segmenting long words, e.g., named-entities. Experimental results show that this is indeed the case. With this improvement, evaluations on the data of the second SIGHAN CWS bakeoff show that our system is competitive with the best ones in the literature.
That Slepen Al the Nyght with Open Ye! Cross-era Sequence Segmentation with Switch-memory

Xuemei Tang,Qi Su,Jun Wang

DOI: https://doi.org/10.18653/v1/2022.acl-long.540

2022-09-07

Abstract:The evolution of language follows the rule of gradual change. Grammar, vocabulary, and lexical semantic shifts take place over time, resulting in a diachronic linguistic gap. As such, a considerable amount of texts are written in languages of different eras, which creates obstacles for natural language processing tasks, such as word segmentation and machine translation. Although the Chinese language has a long history, previous Chinese natural language processing research has primarily focused on tasks within a specific era. Therefore, we propose a cross-era learning framework for Chinese word segmentation (CWS), CROSSWISE, which uses the Switch-memory (SM) module to incorporate era-specific linguistic knowledge. Experiments on four corpora from different eras show that the performance of each corpus significantly improves. Further analyses also demonstrate that the SM can effectively integrate the knowledge of the eras into the neural network.

Computation and Language
Domain-Aware Word Segmentation for Chinese Language: A Document-Level Context-Aware Model

Kaiyu Huang,Keli Xiao,Fengran Mo,Bo Jin,Zhuang Liu,Degen Huang

DOI: https://doi.org/10.1145/3481298

2022-03-31

Abstract:Word segmentation is an essential and challenging task in natural language processing, especially for the Chinese language due to its high linguistic complexity. Existing methods for Chinese word segmentation, including statistical machine learning methods and neural network methods, usually have good performance in specific knowledge domains. Given the increasing importance of interdisciplinary and cross-domain studies, one of the challenges in cross-domain word segmentation is to handle the out-of-vocabulary (OOV) words. Existing methods show unsatisfactory performance to meet the practical standard. To this end, we propose a document-level context-aware model that can automatically perceive and identify OOV words from different domains. Our method jointly implements a word-based and a character-based model and then processes the results with a newly proposed reconstruction model. We evaluate the new method by designing and conducting comprehensive experiments on two real-world datasets (e.g., news from different domains). The results demonstrate the superiority of our method over the state-of-the-art models in handling texts from different domains. Importantly, when doing the word segmentation under the cross-domain scenario, our proposed method can improve the performance of OOV words recognition.

computer science, artificial intelligence

Adaptive Chinese Word Segmentation with Online Passive-Aggressive Algorithm.

The CIPS-SIGHAN CLP 2012 ChineseWord Segmentation Onmicroblog Corpora Bakeoff.

The CIPS-SIGHAN CLP 2014 Chinese Word Segmentation Bake-off.

Addressing Domain Adaptation for Chinese Word Segmentation with Global Recurrent Structure.

A New Error-driven Learning Approach for Chinese Word Segmentation

A realistic and robust model for Chinese word segmentation

Adapting Conventional Chinese Word Segmenter for Segmenting Micro-blog Text: Combining Rule-based and Statistic-based Approaches.

Improving Cross-Domain Chinese Word Segmentation with Word Embeddings

Chinese Word Segmentation with Heterogeneous Graph Neural Network

Automatic Corpus Expansion for Chinese Word Segmentation by Exploiting the Redundancy of Web Information.

Probabilistic Chinese word segmentation with non-local information and stochastic training

Feature Abstraction for Lightweight and Accurate Chinese Word Segmentation

An Improved CRF Based Chinese Language Processing System for SIGHAN Bakeoff 2007.

A Hybrid Approach to Chinese Word Segmentation around CRFs

Toward Fast and Accurate Neural Chinese Word Segmentation with Multi-Criteria Learning

A Long Dependency Aware Deep Architecture for Joint Chinese Word Segmentation and POS Tagging.

Adversarial Multi-Criteria Learning for Chinese Word Segmentation

Chinese Word Segmentation Without Using Lexicon and Hand-Crafted Training Data

A Discriminative Latent Variable Chinese Segmenter with Hybrid Word/Character Information.

That Slepen Al the Nyght with Open Ye! Cross-era Sequence Segmentation with Switch-memory

Domain-Aware Word Segmentation for Chinese Language: A Document-Level Context-Aware Model