Abstract:Word segmentation has been shown helpful for Chinese-to-English machine translation (MT), yet the way different segmentation strategies affect MT is poorly understood. In this paper, we focus on comparing different segmentation strategies in terms of machine translation quality. Our empirical study covers both English-to-Chinese and Chinese-to-English translation for the first time. Our results show the necessity of word segmentation depends on the translation direction. After comparing two types of segmentation strategies with associated linguistic resources, we demonstrate that optimizing segmentation itself does not guarantee better MT performance, and segmentation strategy choice is not the key to improve MT. Instead, we discover that linguistical resources such as segmented corpora or the dictionaries that segmentation tools rely on actually determine how word segmentation affects machine translation. Based on these findings, we propose an empirical approach that directly optimize dictionary with respect to the MT task for word segmenter, providing a BLEU score improvement of 1.30.

Research on Maximum-Matching Algorithm of Chinese Word Segmentation Arithmetic

Improved Algorithm for Increasing Maximum Matching

The Research of Chinese Word Segmentation Algorithm Based on Forward Maximum Match

Research of Reverse Backtracking Matching Algorithm for Chinese Word Segmentation

Chinese Word Segmentation Method Based on Dictionary and Frequency of the Words

Research on high-performance pattern matching algorithm

A Chinese word segmentation system design based on forward-backward maximum matching algorithm

Word Chain-based Automatic Word Segmentation Method

Chinese Analyzer for Search Engine-Lucene

Improved forward algorithm for maximum matching word segmentation

Research on Chinese Automatic Segmentation

A Design and Test of Chinese Word Segmentation Based on Lucene

Research of Chinese Word Segmentation Based on Double-Array Trie

A Pragmatic Approach to Increase Accuracy of Chinese Word-Segmentation

Effective Subsequence-Based Tagging for Chinese Word Segmentation

Chinese Word Segmentation: A Decade Review

Algorithm for Solving 3-Character Crossing Ambiguities in Chinese Word Segmentation

An empirical study on word segmentation for chinese machine translation

Method of Chinese Word Segmentation Based on Directed Graph

A Comparative Study on Chinese Word Segmentation Using Statistical Models

Chinese Word Segmentation with Maximum Entropy and N-gram Language Model