Abstract:Deep contextualized embeddings, as learned by large pre-training models, have proven highly effective in various downstream natural language processing tasks. However, the embedding space in these large models lacks explicit regularization, leading to underfitting and substantial costs during large-scale training on huge corpora. In this paper, we present a novel approach to learning deep contextualized embeddings, introducing linguistic knowledge regularization. Specifically, our proposed model, MorPhEMe (Morphology and Phonology Embedding Memory), features an external addressable memory with two additional addressable memories for storing morphology and phonology knowledge. MorPhEMe can be seamlessly stacked into a deep architecture. Notably different from existing pre-training models, MorPhEMe boasts two distinctive features: (1) compositional encoding and decompositional decoding facilitated by a dynamic addressing mechanism; and (2) explicit memory embedding regularization through cross-layer memory sharing. Theoretical analysis suggests that the inclusion of morphology and phonology enables MorPhEMe to reduce the modeling complexity of natural language sequences. We evaluate MorPhEMe across a diverse set of Chinese natural language processing tasks, including language modeling, word similarity computation, word analogy reasoning, relation extraction, and machine reading comprehension. Experimental results demonstrate that MorPhEMe, in contrast to state-of-the-art models, achieves remarkable improvements with fewer parameters and rapid convergence.

Learning Effective Word Embedding Using Morphological Word Similarity

Co-learning of Word Representations and Morpheme Representations.

Co-learning of Word Representations and Morpheme Representations

KNET: A General Framework for Learning Word Embedding using Morphological Knowledge

Learning word embedding using morphological knowledge

Knowledge-Powered Deep Learning for Word Embedding

Improving Word Embeddings by Emphasizing Co-hyponyms.

Bridging the Semantic Latent Space Between Brain and Machine: Similarity is All You Need

An Exploration Of Semantic Relations In Neural Word Embeddings Using Extrinsic Knowledge

Lexical semantics enhanced neural word embeddings

A Survey of Word Embeddings Based on Deep Learning

Exploring Chinese Word Embedding with Similar Context and Reinforcement Learning

The MorPhEMe Machine: an Addressable Neural Memory for Learning Knowledge-Regularized Deep Contextualized Chinese Embedding

Flexible margins and multiple samples learning to enhance lexical semantic similarity

Morphological Priors for Probabilistic Neural Word Embeddings

Enhancing Semantic Word Representations by Embedding Deeper Word Relationships

An Adaptive Wordpiece Language Model For Learning Chinese Word Embeddings

Learning Semantic Representations for Novel Words: Leveraging Both Form and Context

Learning word embeddings from dependency relations

Training and Evaluating Improved Dependency-Based Word Embeddings.

Learning Word Sense Embeddings from Word Sense Definitions