Abstract:As the demand for global information increases significantly, multilingual corpora has become a valuable linguistic resource for applications to cross‐lingual information retrieval and natural language processing. In order to cross the boundaries that exist between different languages, dictionaries are the most typical tools. However, the general‐purpose dictionary is less sensitive in both genre and domain. It is also impractical to manually construct tailored bilingual dictionaries or sophisticated multilingual thesauri for large applications. Corpus‐based approaches, which do not have the limitation of dictionaries, provide a statistical translation model with which to cross the language boundary. There are many domain‐specific parallel or comparable corpora that are employed in machine translation and cross‐lingual information retrieval. Most of these are corpora between Indo‐European languages, such as English/French and English/Spanish. The Asian/Indo‐European corpus, especially English/Chinese corpus, is relatively sparse. The objective of the present research is to construct English/Chinese parallel corpus automatically from the World Wide Web. In this paper, an alignment method is presented which is based on dynamic programming to identify the one‐to‐one Chinese and English title pairs. The method includes alignment at title level, word level and character level. The longest common subsequence (LCS) is applied to find the most reliable Chinese translation of an English word. As one word for a language may translate into two or more words repetitively in another language, the edit operation, deletion, is used to resolve redundancy. A score function is then proposed to determine the optimal title pairs. Experiments have been conducted to investigate the performance of the proposed method using the daily press release articles by the Hong Kong SAR government as the test bed. The precision of the result is 0.998 while the recall is 0.806. The release articles and speech articles, published by Hongkong & Shanghai Banking Corporation Limited, are also used to test our method, the precision is 1.00, and the recall is 0.948.

Chinese Word Frequency Approximation Based on Multitype Corpora.

Word frequency approximation for chinese using raw, MM-Segmented and manually segmented corpora

Word frequency approximation for chinese without using manually-annotated corpus

Chinese Word Segmentation Method Based on Dictionary and Frequency of the Words

The Structural Complexity of Chinese Words and Its Relationship with Word Frequency.

Extract Chinese Unknown Words from a Large-scale Corpus Using Morphological and Distributional Evidences.

A morphology-based Chinese word segmentation method

A Pragmatic Approach for Classical Chinese Word Segmentation.

A Unified Model for Joint Chinese Word Segmentation and POS Tagging with Heterogeneous Annotation Corpora.

Reducing Approximation and Estimation Errors for Chinese Lexical Processing with Heterogeneous Annotations

A Comparative Study on Chinese Word Segmentation Using Statistical Models

Combining Statistical Model and Dictionary for Domain Adaption of Chinese Word Segmentation

Chinese Word Segmentation Without Using Lexicon and Hand-Crafted Training Data

A Pragmatic Approach to Increase Accuracy of Chinese Word-Segmentation

Discriminate Chinese Word Segmenter with Global and Context Features

Automatic Extraction of Multiword Expressions Combining Statistical and Similarity Approaches

Chinese word segmenter based on discriminative classifiers integration

Automatic Corpus Expansion for Chinese Word Segmentation by Exploiting the Redundancy of Web Information.

Automatic construction of English/Chinese parallel corpora

Experiments on the Use of Corpus-Based Word BI-gram in Chinese Word Segmentation.

Chinese Word Segmentation Probability Dictionary Training and Enrich Solution