The Problem of Separating Characters into Fewest Words and Its Algorithms

XL WANG,KZ WANG,ZR LI,XH BAI

1989-01-01

Chinese Science Bulletin

Abstract:A Chinese word consists of one or several characters. Unlike English, there is no space between Chinese characters. To solve the problem of SCW (separating characters into words) is an important task in the automation of Chinese information processing.

What problem does this paper attempt to address?

Off- Line Chinese Writer Identification Based on Character-Level Decision Combination

Wei Deng,Qinghu Chen,Yucheng Yan,Chunxiao Wan

DOI: https://doi.org/10.1109/ISIP.2008.135

2008-01-01

Abstract:In this paper, a novel method to accelerate off-line Chinese writer identification by combining multi text-sensitive features and combining multi character level decisions is practiced. The used text-sensitive writer identification algorithm extracts Directional histogram feature, Moment feature and Wigner feature, reduces the dimensions using PCA and LDA, and adopts the simple Euclidean classifier. However the writer identification algorithms are text-sensitive, thus there are different mutual character combinations between writers. A method for retrieving in an amount of writers who has different handwriting script content is proposed in this paper, by combining and sorting the posterior probability measures of writing identification. An experiment, which is carried out in a handwriting script database, demonstrates the effectiveness of the proposed method.
Chinese Word Segmentation Method Based on Dictionary and Frequency of the Words

ZHANG HENG,YANG WENZHAO,QU JINGHUI,LU HONGBING,ZHANG LIANG,ZHAO FEI

DOI: https://doi.org/10.3969/j.issn.1008-0570.2008.03.095

2008-01-01

Abstract:Chinese word segmentation is the precondition and base of the Chinese information processing.In the Chinese sentences,it has no obvious interval mark between words.It has polyphony and multi-vocal words in Chinese sentences.So,the result of segmentation unavoidable contains ambiguous words.Chinese segmentation is the "bottleneck" of the Chinese information processing.This article uses dictionary and the frequency of the word to decompose the Chinese sentence,and realizes Chinese auto-segmentation,and uses the methods of computing the frequency of words and ambiguous word eliminating to improve the accuracy of word segmentation
New Word Identification in Social Network Text Based on Time Series Information

Meng Wang,Lanfen Lin,Feng Wang

DOI: https://doi.org/10.1109/cscwd.2014.6846904

2014-01-01

Abstract:Different from the languages widely used in western countries such as English or French, there are no spaces between words in Chinese language, and a segmentation of the texts is necessary before other superior processes. New word identification is an important problem in the segmentation process, especially when the segmentation targets are social network texts which have more abbreviated words or other non-standard representations. Several methods have been proposed to detect Chinese new words. Most of these methods take the corpus as a static set and they don't consider the time domain information. Different from these studies, we regard our social network corpus as a text series spreading along the time line and design a new kind of features named dynamic features which can reflect the temporal variety of the string's statistical features. The experimental results on the dataset crawled from the biggest microblogging application in China show that this method can significantly improve the effect of Chinese new word identification.
Research on Automatic Recognition of Separable Words in Modern Chinese

Bo Liu,Xue-dong Tian,Xin Fu Li

DOI: https://doi.org/10.4028/www.scientific.net/AMM.670-671.1493

2014-10-01

Abstract:Separable words have important applications in many fields such as Chinese information processing, Chinese-English translation, teaching Chinese as a foreign language. There are about five thousand separable words distribute in the corpus of Chinese, and the word frequency is greater in the novel, so the study on identification of separable words is significant. This paper selects the higher discrete frequency of verb-object separable words as the object of the study, by examining the manifestation of extended components in different separable words and giving summary and detailed classification of the extended components on the large-scale corpus, a new approach is designed based on the words segmentation and the structure type of extended component. According to the experiments of identification mark to separable words of verb-object type, the average recall is 89.54% and the average precision is 87.43% in open test. The experimental results show that the method is effective.

Mathematics,Linguistics,Computer Science
Chinese Word Segmentation with Character Abstraction.

Le Tian,Xipeng Qiu,Xuanjing Huang

DOI: https://doi.org/10.1007/978-3-642-41491-6_4

2013-01-01

Abstract:Chinese word segmentation is an important and necessary problem to analyze Chinese texts. In this paper, we focus on the primary challenges in Chinese word segmentation: low accuracy of out-of-vocabulary word. To resolve this difficult problems, we group the "similar" characters to generate more abstract representation. Experimental results show that character abstraction yields a significant relative error reduction of 24.83% in average over the state-of-the-art baseline.
Feature Abstraction for Lightweight and Accurate Chinese Word Segmentation

Le Tian,Xipeng Qiu,Xuanjing Huang

DOI: https://doi.org/10.1109/ialp.2013.65

2013-01-01

Abstract:Chinese word segmentation (CWS) is an important and necessary problem to analyze Chinese texts. The state-of-art CWS systems are mostly based on sequence labeling algorithm and use the discriminative model with millions of overlapping binary features. However, there are few works on porting these systems to the devices with limited computing capacity and memory. In this paper, we focus on two challenges in Chinese word segmentation: (1) low accuracy of out-of-vocabulary word and (2) huge feature space. To resolve these two difficult problems, we propose a method to abstract the original input on both character and feature levels. We group the ``similar'' features to generate more abstract representation. Experimental results show that feature abstraction can greatly reduce the feature space with a comparable performance.
Co-word Analysis using the Chinese Character Set

Loet Leydesdorff,Ping Zhou

DOI: https://doi.org/10.48550/arXiv.0911.1451

2009-11-08

Abstract:Until recently, Chinese texts could not be studied using co-word analysis because the words are not separated by spaces in Chinese (and Japanese). A word can be composed of one or more characters. The online availability of programs that separate Chinese texts makes it possible to analyze them using semantic maps. Chinese characters contain not only information, but also meaning. This may enhance the readability of semantic maps. In this study, we analyze 58 words which occur ten or more times in the 1652 journal titles of the China Scientific and Technical Papers and Citations Database. The word occurrence matrix is visualized and factor-analyzed.

Computation and Language,Digital Libraries
A Compression-based Algorithm for Chinese Word Segmentation

W. J. Teahan,Yingying Wen,Rodger McNab,Ian H. Witten

DOI: https://doi.org/10.1162/089120100561746

IF: 7.778

2000-09-01

Computational Linguistics

Abstract:Chinese is written without using spaces or other word delimiters. Although a text may be thought of as a corresponding sequence of words, there is considerable ambiguity in the placement of boundaries. Interpreting a text as a sequence of words is beneficial for some information retrieval and storage tasks: for example, full-text search, word-based compression, and keyphrase extraction. We describe a scheme that infers appropriate positions for word boundaries using an adaptive language model that is standard in text compression. It is trained on a corpus of presegmented text, and when applied to new text, interpolates word boundaries so as to maximize the compression obtained. This simple and general method performs well with respect to specialized schemes for Chinese language segmentation.

computer science, artificial intelligence, interdisciplinary applications,linguistics
Algorithm for Solving 3-Character Crossing Ambiguities in Chinese Word Segmentation

SUN Maosong,ZUO Zhengping,HUANG Changning

DOI: https://doi.org/10.3321/j.issn:1000-0054.1999.05.027

1999-01-01

Abstract:The technique of Chinese word segmentation plays an important role in many applications of Chinese information processing. Being one of the major types of segmentation ambiguities, crossing ambiguities with length of 3 characters can be frequently found in Chinese running texts. An algorithm aiming at this type of ambiguities is proposed in the paper: instead of making use of part of speech statistical information which needs comparatively high training cost, the algorithm simply employs word frequency information and some common Chinese character subsets with defined properties. The preliminary experiment on 5367 examples, extracted from a Chinese corpus of 0.6 million characters, shows that the segmentation precision of the algorithm reaches 92.07%, which is satisfactory for practical Chinese information processing systems.
Survey on Chinese Word Segmentation

Ying Liu,Danping Li,Xiaolu Zhou,Shuai Xu

DOI: https://doi.org/10.1145/3650215.3650217

2023-01-01

Abstract:With the rapid development of information technique, the amount of text is expanding, which puts forward higher requirements for the accuracy and the speed of Chinese word segmentation. Word segmentation technology is widely used in search engines, AI customer service, man-machine dialogue and other scenarios. This paper summarizes the existing research and challenges. Finally, we looks forward to the possible research directions in the future, and provides a large number of references.
A Statistical Approach For Resolving Problematical Word Boundaries In Chinese Lexicography

oi yee kwong,benjamin k tsou

DOI: https://doi.org/10.1109/ICSMC.2001.972882

2001-01-01

Abstract:Word segmentation is an all important topic in Chinese language processing. Although state-of-the-art segmentation algorithms demonstrate that more than 90% accuracy could possibly be achieved, there remains a subtle question of what constitutes a Chinese word. In this paper, we focus on particular two-character word strings which often raise doubts even for lexicographers as to whether the two characters should be segmented or kept as one word. We experiment with the feasibility of modelling human judgement on such problematical word boundaries by corpus-based mutual information. Preliminary results show that the strength of correlation between the two measures might be lexically as well as structurally dependent, and mutual information only partially model human judgement on problematic Chinese word boundaries.
Towards Unified Chinese Segmentation Algorithm

Fu Lee Wang,Xiaotie Deng,Feng Zou

2006-01-01

Language Resources and Evaluation

Abstract:As Chinese is an ideographic character-based language, the words in the texts are not delimited by spaces. Indexing of Chinese documents is impossible without a proper segmentation algorithm. Many Chinese segmentation algorithms have been proposed in the past. Traditional segmentation algorithms cannot operate without a large dictionary or a large corpus of training data. Nowadays, the Web has become the largest corpus that is ideal for Chinese segmentation. Although the search engines do not segment texts into proper words, they maintain huge databases of documents and frequencies of character sequences in the documents. Their databases are important potential resources for segmentation. In this paper, we propose a segmentation algorithm by mining web data with the help from search engines. It is the first unified segmentation algorithm for Chinese language from different geographical areas. Experiments have been conducted on the datasets of a recent Chinese segmentation competition. The results show that our algorithm outperforms the traditional algorithms in terms of precision and recall. Moreover, our algorithm can effectively deal with the problem of segmentation ambiguity, new word (unknown word) detection, and stop words.
Word Segmentation as Graph Partition.

Yuanhao Liu,Sheng Yu

2018-01-01

Abstract:We propose a new approach to the Chinese word segmentation problem that considers the sentence as an undirected graph, whose nodes are the characters. One can use various techniques to compute the edge weights that measure the connection strength between characters. Spectral graph partition algorithms are used to group the characters and achieve word segmentation. We follow the graph partition approach and design several unsupervised algorithms, and we show their inspiring segmentation results on two corpora: (1) electronic health records in Chinese, and (2) benchmark data from the Second International Chinese Word Segmentation Bakeoff.
Research on Chinese Automatic Segmentation

Wang Fanxiu,Wang Ziqiang

DOI: https://doi.org/10.3969/j.issn.1672-9722.2008.11.016

2008-01-01

Abstract:Word Segmentation is a fundamental problem of the Chinese natural language progressing.Based on the theory of the Chinese word segmentation,an overview of the Chinese word segmentation technology is presented,through the existent Chinese word segmentation system is analyzed,and put forward the greatly improve out-of-vocabulary word recognition method must be development trends of future Chinese automatic segmentation research is put forward.In the end it discuss various factors of Chinese automatic segmentation.
Chinese word segmentation at Peking University

Duan Huiming,Bai Xiaojing,Chang Baobao,Yu Shiwen

DOI: https://doi.org/10.3115/1119250.1119272

2003-01-01

Abstract:Word segmentation is the first step in Chinese information processing, and the performance of the segmenter, therefore, has a direct and great influence on the processing steps that follow. Different segmenters will give different results when handling issues like word boundary. And we will present in this paper that there is no need for an absolute definition of word boundary for all segmenters, and that different results of segmentation shall be acceptable if they can help to reach a correct syntactic analysis in the end.
Chinese Text Classification Using Key Characters String Kernel

Zheng S,Yang Y,Wu H,Liu W.

DOI: https://doi.org/10.1109/SKG.2009.59

2009-01-01

Abstract:Most Chinese text classification methods are based on Chinese word segmentation and bag of words (BOW). The classification performance largely relies on the accuracy of segmentation. Unfortunately, perfect precision and disambiguation of segmentation cannot be reached. In order to solve this problem, a novel Chinese text classification method using string kernel is presented. String kernel computes the similarity of a pair of documents by comparing common substrings they have. Experimental results show that our method greatly enhances the classification on small training data sets. Although the performance of traditional string kernel is comparable to that of BOW methods on larger data set, the dimension of feature space is so high that the calculation process is very time-consuming. Our proposed key characters string kernel technique solves the efficiency and effectiveness problems. Experiments on larger data set show that SVM with key characters string kernel can achieve superior performance.
A Unicode Based Adaptive Segmentor

Q. Lu,S. T. Chan,R. F. Xu,T. S. Chiu,B. L. Li,S. W. Yu

DOI: https://doi.org/10.3115/1119250.1119275

2004-01-01

Abstract:This paper presents a Unicode based Chinese word segmentor. It can handle Chinese text in Simplified, Traditional, or mixed mode. The system uses the strategy of divide-and-conquer to handle the recognition of personal names, numbers, time and numerical values, etc in the preprocessing stage. The segmentor further uses tagging information to work on disambiguation. Adopting a modular design approach, different functional parts are separately implemented using different modules and each module tackles one problem at a time providing more flexibility and extensibility. Results show that with added pre-processing modules and accessorial modules, the accuracy of the segmentor is increased and the system is easily adaptive to different applications.
A new algorithm separating text strings from map images

Xu, T.,Xinggang Lin

DOI: https://doi.org/10.1109/ICDAR.1993.395589

1993-01-01

Abstract:To enter geographic data into computers, people desire to have a system which can scan a paper-based map and feed the raster map image into a computer. Then the computer should automatically extract the geographic information from the map image by image analysis and understanding techniques. A new algorithm which can separate text strings of various fonts, sizes, and styles from other graphic data (line drawing) in the map image is proposed. Numerals are automatically distinguished from Chinese characters based on curved strokes. Disconnected components or parts in a Chinese character can be grouped into meaningful character strings. Experimental results of several test images show the effectiveness of this algorithm
A Hybrid Word-Character Approach to Abstractive Summarization

Chieh-Teng Chang,Chi-Chia Huang,Chih-Yuan Yang,Jane Yung-Jen Hsu

DOI: https://doi.org/10.48550/arXiv.1802.09968

2018-09-08

Abstract:Automatic abstractive text summarization is an important and challenging research topic of natural language processing. Among many widely used languages, the Chinese language has a special property that a Chinese character contains rich information comparable to a word. Existing Chinese text summarization methods, either adopt totally character-based or word-based representations, fail to fully exploit the information carried by both representations. To accurately capture the essence of articles, we propose a hybrid word-character approach (HWC) which preserves the advantages of both word-based and character-based representations. We evaluate the advantage of the proposed HWC approach by applying it to two existing methods, and discover that it generates state-of-the-art performance with a margin of 24 ROUGE points on a widely used dataset LCSTS. In addition, we find an issue contained in the LCSTS dataset and offer a script to remove overlapping pairs (a summary and a short text) to create a clean dataset for the community. The proposed HWC approach also generates the best performance on the new, clean LCSTS dataset.

Computation and Language
Automatic Recognition of Verb-Complement Separable Words Based on BCC

Yuxin Peng,Jian Wu

DOI: https://doi.org/10.1007/978-3-031-28956-9_30

2023-01-01

Abstract:This research is aimed to summarize rules to automatically recognize verb-complement separable words from the perspective of information processing. First, the research based on the BLCU Corpus Center (Abbreviated BCC throughout), uses its retrieval function for verb-complement separable words to perform an exhaustive search for all of them in The Contemporary Chinese Dictionary, and calculate their Positive Rate of identification. Second, summarize and count the internal insertion components of verb-complement separable words in tens of thousands of corpus, observe the before and after components, and conclude them into the corresponding separated form. Research points out the reasons why the corpus recognizes them as verb-complement separable words incorrectly: the result of the machine’s segmentation is wrong, the separated form of verb-object separable words is recognized as a separated form of verb-complement separable words, a morpheme in a compound directional verb is fetched alone, two independent verbs are seen as a separated form, the overlapping usage of verb-complement separable words can’t be recognized accurately, and the ability to deal with polysemous and polysyllabic words is not strong. Finally, transform the separated form from the perspective of linguistics into the corresponding rules that can be recognized by the computer, and write the program for experimentation. This program can make further judgments based on the separable word retrieval of the corpus itself, and realize that the recognition accuracy of these separated words can be increased to more than 90%.

The Problem of Separating Characters into Fewest Words and Its Algorithms

Off- Line Chinese Writer Identification Based on Character-Level Decision Combination

Chinese Word Segmentation Method Based on Dictionary and Frequency of the Words

New Word Identification in Social Network Text Based on Time Series Information

Research on Automatic Recognition of Separable Words in Modern Chinese

Chinese Word Segmentation with Character Abstraction.

Feature Abstraction for Lightweight and Accurate Chinese Word Segmentation

Co-word Analysis using the Chinese Character Set

A Compression-based Algorithm for Chinese Word Segmentation

Algorithm for Solving 3-Character Crossing Ambiguities in Chinese Word Segmentation

Survey on Chinese Word Segmentation

A Statistical Approach For Resolving Problematical Word Boundaries In Chinese Lexicography

Towards Unified Chinese Segmentation Algorithm

Word Segmentation as Graph Partition.

Research on Chinese Automatic Segmentation

Chinese word segmentation at Peking University

Chinese Text Classification Using Key Characters String Kernel

A Unicode Based Adaptive Segmentor

A new algorithm separating text strings from map images

A Hybrid Word-Character Approach to Abstractive Summarization

Automatic Recognition of Verb-Complement Separable Words Based on BCC