Attention-Based Multi-Layer Chinese Word Embedding

Bing Ma,Haifeng Sun,Jingyu Wang,Qi Qi
DOI: https://doi.org/10.1109/BigData47090.2019.9006279
2019-01-01
Abstract:Word embedding is a basic task in natural language processing area. Unlike English, Chinese subword units, such as characters, radicals, and components, contain rich semantic information which can be used to enhance word embeddings. However, existing methods neglect the semantic contribution of corresponding subword units to the word. In this work, we employ attention mechanism to capture the semantic structure of Chinese words and propose a novel framework, named Attention-based multi-Layer Word Embedding model(ALWE). We also design an asynchronous strategy for updating embedding arid attention efficiently. Our model learns to share subword information between distinct words selectively and adaptively. Experimental results on the word similarity, word analogy, and text classification show that the proposed model outperforms all baselines, especially when words don't appear frequently. Qualitative analysis further demonstrates the superiority of ALWE.
What problem does this paper attempt to address?