Abstract:AbstractUnlike English and other Western languages, many Asian languages such as Chinese and Japanese do not delimit words by space. Word segmentation and new word detection are therefore key steps in processing these languages. Chinese word segmentation can be considered as a part-of-speech (POS)-tagging problem. We can segment corpus by assigning a label for each character which indicates the position of the character in a word (e.g., “B” for word beginning, and “E” for the end of the word, etc.). Chinese word segmentation seems to be well studied. Machine learning models such as conditional random field (CRF) and bi-directional long short-term memory (LSTM) have shown outstanding performances on this task. However, the segmentation accuracies drop significantly when applying the same approaches to out-domain cases, in which high-quality in-domain training data are not available. An example of out-domain applications is the new word detection in Chinese microblogs for which the availability of high-quality corpus is limited. In this paper, we focus on out-domain Chinese new word detection. We first design a new method Edge Likelihood (EL) for Chinese word boundary detection. Then we propose a domain-independent Chinese new word detector (DICND); each Chinese character is represented as a low-dimensional vector in the proposed framework, and segmentation-related features of the character are used as the values in the vector.

Joint Model Using Character and Word Embeddings for Detecting Internet Slang Words

Do Multi-Sense Embeddings Improve Natural Language Understanding?

Slang Feature Extraction by Analysing Topic Change on Social Media

Joint Learning of Character and Word Embeddings.

A Study of Slang Representation Methods

Slang Analysis Based on Variant Information Extraction Focusing on the Time Series Topics

A Joint Model for Word Embedding and Word Morphology

Combining Word-Level and Character-Level Representations for Relation Classification of Informal Text

SLANG: New Concept Comprehension of Large Language Models

A quantitative semantic study on Chinese-Japanese homographs based on word embedding

Joint-Character-Poc N-Gram Language Modeling for Chinese Speech Recognition

EmojiLM: Modeling the New Emoji Language

Learning Chinese-Japanese Bilingual Word Embedding by Using Common Characters.

Joint Semantic Synthesis and Morphological Analysis of the Derived Word

Combining Character-Level Representation For Relation Classification

Toward Informal Language Processing: Knowledge of Slang in Large Language Models

Joint Emoji Classification And Embedding Learning

Knowledge of Words: an Interpretable Approach for Personality Recognition from Social Media

Out-domain Chinese New Word Detection with Statistics-Based Character Embedding

Joint Word Representation Learning using a Corpus and a Semantic Lexicon

Pre-Trained Joint Model for Intent Classification and Slot Filling with Semantic Feature Fusion