Abstract:Words provide a useful source of information for Chinese NLP, and word segmentation has been taken as a pre-processing step for most downstream tasks. For many NLP tasks, however, word segmentation can introduce noise and lead to error propagation. The rise of neural representation learning models allows sentence-level semantic information to be collected from characters directly. As a result, it is an empirical question whether a fully character-based model should be used instead of first performing word segmentation. We investigate a neural representation that simultaneously encodes character and word information without the need for segmentation. In particular, candidate words are found in a sentence by matching with a pre-defined lexicon. A lattice structured LSTM is used to encode the resulting word-character lattice, where gate vectors are used to control information flow through words, so that the more useful words can be automatically identified by end-to-end training. We compare the performance of the resulting lattice LSTM and baseline sequence LSTM structures over both character sequences and automatically segmented word sequences. Results on NER show that the character-word lattice model can significantly improve the performance. In addition, as a general sentence representation architecture, character-word lattice LSTM can also be used for learning contextualized representations. To this end, we compare lattice LSTM structure with its sequential LSTM counterpart, namely ELMo. Results show that our lattice version of ELMo gives better language modeling performances. On Chinese POS-tagging, chunking and syntactic parsing tasks, the resulting contextualized Chinese embeddings also give better performance than ELMo trained on the same data.

Joint-Character-Poc N-Gram Language Modeling for Chinese Speech Recognition

Joint n-gram Chinese language modeling with an application to Chinese word segmentation

Six-Writings multimodal processing with pictophonetic coding to enhance Chinese language models

Toward Joint Language Modeling for Speech Units and Text

Lattice LSTM for Chinese Sentence Representation

A Word Language Model Based Contextual Language Processing On Chinese Character Recognition

N-gram Language Model for Chinese Function-word-centered Patterns

A Pair-Based Language Model for the Robust Lexical Analysis in Chinese Text-to-speech Synthesis.

LM Enhanced BiRNN-CRF for Joint Chinese Word Segmentation and POS Tagging.

Character-Level Dependency Model for Joint Word Segmentation, POS Tagging, and Dependency Parsing in Chinese.

Chinese Nested Named Entity Recognition Using a Joint Model

Improving Mandarin End-to-End Speech Recognition with Word N-gram Language Model

Toward a Unified Approach to Statistical Language Modeling for Chinese

Joint Learning of Character and Word Embeddings.

Exploiting Common Characters in Chinese and Japanese to Learn Cross-Lingual Word Embeddings Via Matrix Factorization.

Deep joint learning for language recognition

Exploiting Word Positional Information in Ngram Model for Chinese Text Input Method

Recurrent Neural Network Language Model with Part-of-speech for Mandarin Speech Recognition.

Integrating Language Model in Handwritten Chinese Text Recognition.

Multi-Level Modeling Units for End-to-End Mandarin Speech Recognition

A Joint Model for Unsupervised Chinese Word Segmentation.