Abstract:User-generated text in social media communication (SMC) is mainly characterized by non-standard form. It may contain code switching (CS) text, a widespread phenomenon in SMC, in addition to noisy elements used, especially in written conversations (use of abbreviations, symbols, emoticons) or misspelled words. All of these factors constitute a wall in front of text mining applications. Common text mining tools are dedicated to standard use of standard languages but cannot deal with other forms, especially written text in social media. To overcome these problems, in this work we present our solution for the normalization of non-standard use of standard and non-standard languages (dialects) in SMC text with the use of existent resources and tools. The main processing in our solution consists of CS normalization from multiple to one language by the use of a machine translation--like approach. This processing relies on a linguistic approach of CS, which aims at identifying automatically the translation source and target languages (without human intervention). The remaining processing operations concern the normalization of SMC special expressions and spelling correction of out-of-vocabulary words. To preserve the coded-switched sentence meaning across translation, we adopt a knowledge-based approach for word sense translation disambiguation reinforced with a multi-lingual vertical context. All of these processes are embedded in what we refer to as the machine normalization system. Our solution can be used as a front-end of text mining processing, enabling the analysis of SMC noisy text. The conducted experiments show that our system performs better than considered baselines.

On the performance of phonetic algorithms in microtext normalization

Normalizing Text using Language Modelling based on Phonetics and String Similarity

Machine Normalization

Chinese-English Mixed Text Normalization

Tweet Normalization with Syllables.

What is lost in Normalization? Exploring Pitfalls in Multilingual ASR Model Evaluations

Normalization of Lithuanian Text Using Regular Expressions

MoNoise: Modeling Noise Using a Modular Normalization System

Adapting Sequence to Sequence models for Text Normalization in Social Media

Text Normalization in Chinese Text-to-Speech System

A phonetic-based approach to Chinese chat text normalization

A Phoneme-Scale Assessment of Multichannel Speech Enhancement Algorithms

Don't Touch My Diacritics

Diversity by Phonetics and its Application in Neural Machine Translation

Text Normalization in Mandarin Text-to-speech System.

Normalization of Transliterated Words in Code-Mixed Data Using Seq2Seq Model & Levenshtein Distance

Performance Improvements of Probabilistic Transcript-adapted ASR with Recurrent Neural Network and Language-specific Constraints

A Three-Stage Text Normalization Strategy for Mandarin Text-to-Speech Systems

RNN Approaches to Text Normalization: A Challenge

Voice Conversion Based Speaker Normalization for Acoustic Unit Discovery

Two Approaches to Diachronic Normalization of Polish Texts