Researches on Full Text Retrieval

杨建林

DOI: https://doi.org/10.3969/j.issn.1000-7490.2000.01.005

2000-01-01

Abstract:A new algorithm for automatic segmentation of Chinese word with the stop word list and post-controlled thesaurus, that has absorbed the ideas from the single-Chinese character method and the thesaurus method, is given. Based on this algorithm, a new full text retrieval mode is built.

What problem does this paper attempt to address?

Chinese Word Segmentation Method Based on Dictionary and Frequency of the Words

ZHANG HENG,YANG WENZHAO,QU JINGHUI,LU HONGBING,ZHANG LIANG,ZHAO FEI

DOI: https://doi.org/10.3969/j.issn.1008-0570.2008.03.095

2008-01-01

Abstract:Chinese word segmentation is the precondition and base of the Chinese information processing.In the Chinese sentences,it has no obvious interval mark between words.It has polyphony and multi-vocal words in Chinese sentences.So,the result of segmentation unavoidable contains ambiguous words.Chinese segmentation is the "bottleneck" of the Chinese information processing.This article uses dictionary and the frequency of the word to decompose the Chinese sentence,and realizes Chinese auto-segmentation,and uses the methods of computing the frequency of words and ambiguous word eliminating to improve the accuracy of word segmentation
A Survey of Chinese Word Segmentation and the Application in Information Retrieval

曹倩,丁艳,王超,潘金贵

DOI: https://doi.org/10.3969/j.issn.1001-3695.2004.05.024

2004-01-01

Abstract:Chinese word automatic segment is a essential problem of Chinese Information Retrieval(IR) , as a survey of Chinese word segmentation. In this paper, we introduce the application of it in Chinese IR, then present the work of the word segmentation, analyze the ways and difficulties to combine them.
Research on Chinese Automatic Segmentation

Wang Fanxiu,Wang Ziqiang

DOI: https://doi.org/10.3969/j.issn.1672-9722.2008.11.016

2008-01-01

Abstract:Word Segmentation is a fundamental problem of the Chinese natural language progressing.Based on the theory of the Chinese word segmentation,an overview of the Chinese word segmentation technology is presented,through the existent Chinese word segmentation system is analyzed,and put forward the greatly improve out-of-vocabulary word recognition method must be development trends of future Chinese automatic segmentation research is put forward.In the end it discuss various factors of Chinese automatic segmentation.
Survey on Chinese Word Segmentation

Ying Liu,Danping Li,Xiaolu Zhou,Shuai Xu

DOI: https://doi.org/10.1145/3650215.3650217

2023-01-01

Abstract:With the rapid development of information technique, the amount of text is expanding, which puts forward higher requirements for the accuracy and the speed of Chinese word segmentation. Word segmentation technology is widely used in search engines, AI customer service, man-machine dialogue and other scenarios. This paper summarizes the existing research and challenges. Finally, we looks forward to the possible research directions in the future, and provides a large number of references.
Word Chain-based Automatic Word Segmentation Method

杨建林,张国梁

DOI: https://doi.org/10.3969/j.issn.1000-7490.2000.02.002

2000-01-01

Abstract:An algorithm for automatic segmentation of Chinese word,which is an improved version of the minimum matching algorithm,is put forward.The key idea of the algorithm is to optimize the word bank and the matching process to enhance the speed and accuracy of word segmentation.By integrating the case bank for processing ambiguous word chain with relevant segmentation rules,the correctness of word segmentation is enhanced,which partly makes up the deficiency in processing natural language.
Research of Reverse Backtracking Matching Algorithm for Chinese Word Segmentation

Wei Li,Y. Liu

Abstract:With the rapid development of science and technology, we have entered the age of digital information. Search engines have become a preferred tool to find information, and Chinese word segmentation is an important part of the search engines. In order to improve the efficiency of the existing Chinese word segmentation and solve its existence problems of ambiguity, the reverse backtracking algorithm based on the reverse maximum matching method is designed. The algorithm pretreats the text to be split firstly, then use the forward and reverse maximum matching algorithm to discover ambiguities efficiently, finally use the backtracking mechanism to eliminate ambiguity. Experiments show that this method improves the speed and reduces the error rate of the overlapping ambiguity, improves the performance of the reverse maximum matching algorithm.

Computer Science
Towards Unified Chinese Segmentation Algorithm

Fu Lee Wang,Xiaotie Deng,Feng Zou

2006-01-01

Language Resources and Evaluation

Abstract:As Chinese is an ideographic character-based language, the words in the texts are not delimited by spaces. Indexing of Chinese documents is impossible without a proper segmentation algorithm. Many Chinese segmentation algorithms have been proposed in the past. Traditional segmentation algorithms cannot operate without a large dictionary or a large corpus of training data. Nowadays, the Web has become the largest corpus that is ideal for Chinese segmentation. Although the search engines do not segment texts into proper words, they maintain huge databases of documents and frequencies of character sequences in the documents. Their databases are important potential resources for segmentation. In this paper, we propose a segmentation algorithm by mining web data with the help from search engines. It is the first unified segmentation algorithm for Chinese language from different geographical areas. Experiments have been conducted on the datasets of a recent Chinese segmentation competition. The results show that our algorithm outperforms the traditional algorithms in terms of precision and recall. Moreover, our algorithm can effectively deal with the problem of segmentation ambiguity, new word (unknown word) detection, and stop words.
Auto-Indexing Based on Chinese Characters Coding on Words Platform

JIAO Hui,LIU Qian,JIA Hui-bo

DOI: https://doi.org/10.3321/j.issn:1002-8331.2007.15.053

2007-01-01

Computer Engineering and Applications Journal

Abstract:Auto-indexing is one of the key techniques of information retrieval based on contents.At present the research on Chinese auto-indexing mainly focuses on automatic segmentation which is a predisposal problem.This paper presents a kind of Chinese characters coding method on words platform,and establishes a new Chinese text format in computer which makes words the smallest information unit.Based on this method,auto-indexing does not rely on segmentation as before.Thereby the efficiency and quality of auto-indexing would be improved.
Automatic Construction of Chinese Stop Word List

Feng Zou,Fu Lee Wang,Xiaotie Deng,Song Han,Lu Sheng Wang

2006-01-01

Research in Computing Science

Abstract:In modern information retrieval systems, effective indexing can be achieved by removal of stop words. Till now many stop word lists have been developed for English language. However, no standard stop word list has been constructed for Chinese language yet. With the fast development of information retrieval in Chinese language, exploring Chinese stop word lists becomes critical. In this paper, to save the time and release the burden of manual stop word selection, we propose an automatic aggregated methodology based on statistical and information models for extraction of a stop word list in Chinese language. Result analysis shows that our stop list is comparable with a general English stop word list, and our list is much more general than other Chinese stop lists as well. Our stop word extraction algorithm is a promising technique, which saves the time for manual generation and constructs a standard. It could be applied into other languages in the future.
TopWORDS-Seg: Simultaneous Text Segmentation and Word Discovery for Open-Domain Chinese Texts via Bayesian Inference

Changzai Pan,Maosong Sun,Ke Deng

DOI: https://doi.org/10.18653/v1/2022.acl-long.13

2022-01-01

Abstract:Processing open-domain Chinese texts has been a critical bottleneck in computational linguistics for decades, partially because text segmentation and word discovery often entangle with each other in this challenging scenario. No existing methods yet can achieve effective text segmentation and word discovery simultaneously in open domain. This study fills in this gap by proposing a novel method called TopWORDS-Seg based on Bayesian inference, which enjoys robust performance and transparent interpretation when no training corpus and domain vocabulary are available. Advantages of TopWORDS-Seg are demonstrated by a series of experimental studies.
Automatic summarization oriented Chinese word extraction and statistics system

Hong Guang Suo,Xinxin Zhao,Yatao Duan,Yan Zhang,YuShu Liu

2007-01-01

Journal of Computational Information Systems

Abstract:Automatic word segmentation is the foundation in Chinese text processing. It is a direction that to study different word segmentation method according to different purpose. Word segmentation and statistics are basic procedures in Chinese automatic summarization based on extraction. A pretreatment system is presented which combined word segmentation and statistics and achieved by 'Positional Remembering and Jump Matching with Statistics' in one scanning process. The experimental results show that it has a high speed and better effect in summarization.
An Application Of Lexical Semantics Of Chinese In Webpage Keyword Extraction Algorithm

Chanjuan Wang,Bin Sun,Lu Zhang

2010-01-01

Abstract:The principles of Lexical Semantics provide guidance for webpage keyword extraction. This paper designs an algorithm that utilizes Chinese words' semantic features and leads to decent extraction effects. According to the semantic systematic principle and the mutual dependence among sememes, the algorithm further enhances its performance by introducing a set of "seed keywords", thereby the conceptual feature of the extracted keywords is emphasized, to which advertisers can attach more references.
Research on Chinese Keywords Extraction Based on Characters Sequence Annotation

王昊,邓三鸿,苏新宁

DOI: https://doi.org/10.11925/infotech.1003-3513.2011.12.06

2012-01-01

Abstract:Based on the whole Chinese booklist of a certain university library as well as the analysis of its book indexing information, the paper summarizes the features and extracting laws of Chinese keywords, and establishes a Chinese key- words extraction model based on characters sequence annotation, which proposes the basic idea and implementation scheme for extracting keywords. It verifies the feasibility, rationality and practicality of the model by large - scale experiments, and basically solves the problems of Chinese keywords extraction without executing words segmentation, which shows that characters sequence annotation is better than words sequence annotation.
Parsing-based Chinese word segmentation integrating morphological and syntactic information

Xihong Wu,Meng Zhang,Xiaojun Lin

DOI: https://doi.org/10.1109/NLPKE.2011.6138178

2011-01-01

Abstract:The conventional sequence labeling methods for Chinese word segmentation do not fully utilize the linguistic information, which restricts further improvements of the performance. Chinese morphology intensively investigates the constructions and usages of Chinese words, which is helpful to Chinese word segmentation. Furthermore, some word segmentation ambiguities cannot be resolved only by means of the lexical information, and the final disambiguations take place in the parsing process. In this paper, we propose a parsing-based Chinese word segmentation model, which can fully utilize the morphological and syntactic information. Experiments on Penn Chinese Treebank(CTB) 5.0 show that the proposed model obtains competitive performances as the CRFs-based model. To investigate the relationship between our parsing-based model and the CRFs-based model, a maximum entropy model based framework for integrating different knowledge sources is employed. The integrating model obtains an F-measure of 97.9, 25% in segmentation error rate reduction relative to the CRFs-based model, which indicates that the two models are complementary to each other.
Stop word list construction and application in Chinese language processing

Feng Zou,Fu Lee Wang,Xiaotie Deng,Song Han,Lu Sheng Wang

2006-01-01

WSEAS TRANSACTIONS ON INFORMATION SCIENCE AND APPLICATIONS

Abstract:In modern information retrieval systems, effective indexing can be achieved by removal of stop words. Till now many stop word lists have been developed for English language. However, no standard stop word list has been constructed for Chinese language yet. With the fast development of information retrieval in Chinese language, exploring Chinese stop word lists becomes critical. In this paper, to save the time and release the burden of manual stop word selection, we propose an automatic aggregated methodology based on statistical and information models for extraction of stop word list in Chinese language. Result analysis shows that our stop list is comparable with a general English stop word list, and our list is much more general than other Chinese stop lists as well. Extensive experiments have been conducted on Chinese segmentation to investigate the effectiveness of the stop word list extracted. The …
Research of Chinese Word Segmentation Based on Double-Array Trie

Zhao Huan

Abstract:This paper proposed some improved strategies for the algorithm of Double-Array Trie.Firstly,the priority was given to the node with most child nodes in order to avoid the collision;secondly,an empty-list was defined;Finally,the collision node was added to a hash table,which avoided re-allocation.Then,we implemented a program for a Chinese word segmentation system based on the improved Double-Array Trie and compared it with several other methods.From the results,it turns out that the insertion time and the space efficiency are achieved,and that search efficiency is improved.

Computer Science
Chinese Keyword Extraction Based on Word Platform

Hui Jiao,Qian Liu,Hui-bo Jia

DOI: https://doi.org/10.1109/fskd.2007.215

2007-01-01

Abstract:At present researches on Chinese keyword extraction mainly focus on automatic segmentation which is a pretreatment problem. This paper presents a kind of Chinese encoding method based on word platform, and establishes a new Chinese document format in computer. This method makes word the smallest information unit. Chinese keyword extraction does not rely on segmentation by this new method. Thereby the efficiency and quality could be improved. Statistical analysis is adopted to conduct the experiment of keyword extraction based on word platform, and experimental results are satisfying.
Chinese Text Retrieval: an Adjacency Matrix Based Approach

SG Zhou,YF Hu

DOI: https://doi.org/10.1109/icsmc.2001.972887

2001-01-01

Abstract:This paper explores an adjacency matrix based approach for Chinese text retrieval. Using this approach, retrieval systems need only keep indexing data, rather than indexing data and original text data as traditional retrieval systems do, thus system space costs as a whole can be reduced drastically while retrieval efficiency is maintained as satisfactory. Experiments demonstrate the effectiveness and efficiency of this approach
A Hybrid Approach to Chinese Word Segmentation around CRFs

ZHOU Jun-sheng,DAI Xin-yu,NI Rui-yu,CHEN Jia-jun

2005-01-01

Abstract:In this paper, we present a Chinese word segmentation system which is consisted of four components, i.e. basic segmentation, named entity recognition, error-driven learner and new word detector. The basic segmentation and named entity recognition, implemented based on conditional random fields, are used to generate initial segmentation results. The other two components are used to refine the results. Our system participated in the tests on open and closed tracks of Beijing University (PKU) and Microsoft Research (MSR). The actual evaluation results show that our system performs very well in MSR open track, MSR closed track and PKU open track.
Chinese word segmentation and its effect on information retrieval

Schubert Foo,Hui Li

DOI: https://doi.org/10.1016/s0306-4573(02)00079-1

2004-01-01

Abstract:A set of IR experiments was carried out to study the impact of Chinese word segmentation and its effect on information retrieval (IR) at the Division of Information Studies, Nanyang Technological University, Singapore. A total of four automatic character-based segmentation approaches and a manual word segmentation approach was first carried out to obtain the word segments for indexing and to evaluate the segmentation accuracy of these automatic approaches. The IR experiments study both the influence of different document segmentation approaches on IR effectiveness and the methods used for query segmentation. Traditional data recall and precision measures were used to gauge IR effectiveness. A number of queries were selected and subjected to further detailed analysis to further explore the influence of word segmentation on IR.The findings reveal that the segmentation approach has an effect on IR effectiveness. Better IR results are obtained by using the same method for query and document processing as this increase the probability of the query-document match. The recognition of a higher number of 2-character words generally contributes to the improvement of IR effectiveness. However, manual segmentation does not always work better than character-based segmentation as a result of the existence of longer words with more than two characters. No evidence is found that ambiguous words resulting from the segmentation process significantly affect IR.

computer science, information systems,information science & library science

Researches on Full Text Retrieval

Chinese Word Segmentation Method Based on Dictionary and Frequency of the Words

A Survey of Chinese Word Segmentation and the Application in Information Retrieval

Research on Chinese Automatic Segmentation

Survey on Chinese Word Segmentation

Word Chain-based Automatic Word Segmentation Method

Research of Reverse Backtracking Matching Algorithm for Chinese Word Segmentation

Towards Unified Chinese Segmentation Algorithm

Auto-Indexing Based on Chinese Characters Coding on Words Platform

Automatic Construction of Chinese Stop Word List

TopWORDS-Seg: Simultaneous Text Segmentation and Word Discovery for Open-Domain Chinese Texts via Bayesian Inference

Automatic summarization oriented Chinese word extraction and statistics system

An Application Of Lexical Semantics Of Chinese In Webpage Keyword Extraction Algorithm

Research on Chinese Keywords Extraction Based on Characters Sequence Annotation

Parsing-based Chinese word segmentation integrating morphological and syntactic information

Stop word list construction and application in Chinese language processing

Research of Chinese Word Segmentation Based on Double-Array Trie

Chinese Keyword Extraction Based on Word Platform

Chinese Text Retrieval: an Adjacency Matrix Based Approach

A Hybrid Approach to Chinese Word Segmentation around CRFs

Chinese word segmentation and its effect on information retrieval