ORTHOGRAPHIC CASE RESTORATION USING SUPERVISED LEARNING WITHOUT MANUAL ANNOTATION

CHENG NIU,WEI LI,JIHONG DING,ROHINI K. SRIHARI
DOI: https://doi.org/10.1142/s0218213004001454
IF: 1.1
2004-03-01
International Journal on Artificial Intelligence Tools
Abstract:One challenge in text processing is the treatment of case insensitive documents such as speech recognition results. The traditional approach is to re-train a language model excluding case-related features. This paper presents an alternative two-step approach whereby a preprocessing module (Step 1) is designed to restore case-sensitive form which is subsequently processed by the original system (Step 2). Step 1 is mainly implemented as a Hidden Markov Model trained on a large raw corpus of case sensitive documents. It is demonstrated that this approach (i) outperforms the feature exclusion approach for named entity tagging, (ii) leads to limited degradation for parsing, relationship extraction and case insensitive question answering, (iii) reduces system complexity, and (iv) has wide applicability: the restored text can be used in both statistical model and rule-based systems.
computer science, artificial intelligence, interdisciplinary applications
What problem does this paper attempt to address?