Abstract:Deep contextualized embeddings, as learned by large pre-training models, have proven highly effective in various downstream natural language processing tasks. However, the embedding space in these large models lacks explicit regularization, leading to underfitting and substantial costs during large-scale training on huge corpora. In this paper, we present a novel approach to learning deep contextualized embeddings, introducing linguistic knowledge regularization. Specifically, our proposed model, MorPhEMe (Morphology and Phonology Embedding Memory), features an external addressable memory with two additional addressable memories for storing morphology and phonology knowledge. MorPhEMe can be seamlessly stacked into a deep architecture. Notably different from existing pre-training models, MorPhEMe boasts two distinctive features: (1) compositional encoding and decompositional decoding facilitated by a dynamic addressing mechanism; and (2) explicit memory embedding regularization through cross-layer memory sharing. Theoretical analysis suggests that the inclusion of morphology and phonology enables MorPhEMe to reduce the modeling complexity of natural language sequences. We evaluate MorPhEMe across a diverse set of Chinese natural language processing tasks, including language modeling, word similarity computation, word analogy reasoning, relation extraction, and machine reading comprehension. Experimental results demonstrate that MorPhEMe, in contrast to state-of-the-art models, achieves remarkable improvements with fewer parameters and rapid convergence.

Implanting Rational Knowledge into Distributed Representation at Morpheme Level.

Co-learning of Word Representations and Morpheme Representations.

Co-learning of Word Representations and Morpheme Representations

Learning Distributed Representations Of Uyghur Words And Morphemes

Learning Effective Word Embedding Using Morphological Word Similarity

Knowledge Graph Embedding with Diversity of Structures

KNET: A General Framework for Learning Word Embedding using Morphological Knowledge

Towards a Description of Chinese Morpheme Conceptions and Semantic Composition of Word

Joint Semantic Synthesis and Morphological Analysis of the Derived Word

Improving Chinese Word Representation with Conceptual Semantics

Improved Learning of Chinese Word Embeddings with Semantic Knowledge.

The MorPhEMe Machine: an Addressable Neural Memory for Learning Knowledge-Regularized Deep Contextualized Chinese Embedding

Knowledge-Powered Deep Learning for Word Embedding

Incorporating Linguistic Knowledge for Learning Distributed Word Representations.

A Character-Enhanced Chinese Word Embedding Model.

Lexicon-matched Word Injection for Chinese NER.

Knowledge representation of non-literal meanings of Chinese words and its applications

Semantic Processing Through Distributed Representation of Chinese Words

Combining Word-Level and Character-Level Representations for Relation Classification of Informal Text

Structured Vectors for Chinese Word Representations

A morphology-based Chinese word segmentation method