Improve Chinese Word Embeddings by Exploiting Internal Structure.

Jian Xu,Jiawei Liu,Liangang Zhang,Zhengyu Li,Huanhuan Chen

DOI: https://doi.org/10.18653/v1/n16-1119

2016-01-01

Abstract:Jian Xu, Jiawei Liu, Liangang Zhang, Zhengyu Li, Huanhuan Chen. Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. 2016.

What problem does this paper attempt to address?

Improve Word Embedding Using Both Writing and Pronunciation.

Wenhao Zhu,Xin Jin,Jianyue Ni,Baogang Wei,Zhiguo Lu

DOI: https://doi.org/10.1371/journal.pone.0208785

IF: 3.7

2018-01-01

PLoS ONE

Abstract:Text representation can map text into a vector space for subsequent use in numerical calculations and processing tasks. Word embedding is an important component of text representation. Most existing word embedding models focus on writing and utilize context, weight, dependency, morphology, etc., to optimize the training. However, from the linguistic point of view, spoken language is a more direct expression of semantics; writing has meaning only as a recording of spoken language. Therefore, this paper proposes the concept of a pronunciation-enhanced word embedding model (PWE) that integrates speech information into training to fully apply the roles of both speech and writing to meaning. This paper uses the Chinese language, English language and Spanish language as examples and presents several models that integrate word pronunciation characteristics into word embedding. Word similarity and text classification experiments show that the PWE outperforms the baseline model that does not include speech information. Language is a storehouse of sound-images; therefore, the PWE can be applied to most languages.
Enhanced Double-Carrier Word Embedding Via Phonetics and Writing

Wenhao Zhu,Xin Jin,Shuang Liu,Zhiguo Lu,Wu Zhang,Ke Yan,Baogang Wei

DOI: https://doi.org/10.1145/3344920

IF: 1.471

2020-01-01

ACM Transactions on Asian and Low-Resource Language Information Processing

Abstract:Word embeddings, which map words into a unified vector space, capture rich semantic information. From a linguistic point of view, words have two carriers, speech and writing. Yet the most recent word embedding models focus on only the writing carrier and ignore the role of the speech carrier in semantic expressions. However, in the development of language, speech appears before writing and plays an important role in the development of writing. For phonetic language systems, the written forms are secondary symbols of spoken ones. Based on this idea, we carried out our work and proposed double-carrier word embedding (DCWE). We used DCWE to conduct a simulation of the generation order of speech and writing. We trained written embedding based on phonetic embedding. The final word embedding fuses writing and phonetic embedding. To illustrate that our model can be applied to most languages, we selected Chinese, English, and Spanish as examples and evaluated these models through word similarity and text classification experiments.
Using Context-to-Vector with Graph Retrofitting to Improve Word Embeddings

Jiangbin Zheng,Yile Wang,Ge Wang,Jun Xia,Yufei Huang,Guojiang Zhao,Yue Zhang,Stan Z. Li

DOI: https://doi.org/10.18653/v1/2022.acl-long.561

2022-01-01

Abstract:Although contextualized embeddings generated from large-scale pre-trained models perform well in many tasks, traditional static embeddings (e.g., Skip-gram, Word2Vec) still play an important role in low-resource and lightweight settings due to their low computational cost, ease of deployment, and stability. In this paper, we aim to improve word embeddings by 1) incorporating more contextual information from existing pre-trained models into the Skip-gram framework, which we call Context-to-Vec; 2) proposing a post-processing retrofitting method for static embeddings independent of training by employing priori synonym knowledge and weighted vector distribution. Through extrinsic and intrinsic tasks, our methods are well proven to outperform the baselines by a large margin.
Joint Learning of Character and Word Embeddings.

Xinxiong Chen,Lei Xu,Zhiyuan Liu,Maosong Sun,Huanbo Luan

2015-01-01

Abstract:Most word embedding methods take a word as a basic unit and learn embeddings according to words' external contexts, ignoring the internal structures of words. However, in some languages such as Chinese, a word is usually composed of several characters and contains rich internal information. The semantic meaning of a word is also related to the meanings of its composing characters. Hence, we take Chinese for example, and present a character-enhanced word embedding model (CWE). In order to address the issues of character ambiguity and non-compositional words, we propose multiple prototype character embeddings and an effective word selection method. We evaluate the effectiveness of CWE on word relatedness computation and analogical reasoning. The results show that CWE outperforms other baseline methods which ignore internal character information. The codes and data can be accessed from https://github.com/Leonard-Xu/CWE.
Pronunciation-Enhanced Chinese Word Embedding

Qinjuan Yang,Haoran Xie,Gary Cheng,Fu Lee Wang,Yanghui Rao

DOI: https://doi.org/10.1007/s12559-021-09850-9

IF: 4.89

2021-02-22

Cognitive Computation

Abstract:Abstract Chinese word embeddings have recently garnered considerable attention. Chinese characters and their sub-character components, which contain rich semantic information, are incorporated to learn Chinese word embeddings. Chinese characters can represent a combination of meaning, structure, and pronunciation. However, existing embedding learning methods focus on the structure and meaning of Chinese characters. In this study, we aim to develop an embedding learning method that can make complete use of the information represented by Chinese characters, including phonology, morphology, and semantics. Specifically, we propose a pronunciation-enhanced Chinese word embedding learning method, where the pronunciations of context characters and target characters are simultaneously encoded into the embeddings. Evaluation of word similarity, word analogy reasoning, text classification, and sentiment analysis validate the effectiveness of our proposed method.

computer science, artificial intelligence,neurosciences
Hierarchical Joint Learning for Chinese Word Embeddings

Zhe Wang,Qiong Wang,Yechao Bai,Xinggan Zhang

DOI: https://doi.org/10.1109/icsidp47821.2019.9172912

2019-01-01

Abstract:In recent years, some methods of learning Chinese word embedding based on the characters and sub-characters information have been proposed and achieve state-of-art results. Generally speaking, good embedding of characters and sub-characters can help learn better embedding of words. However, current methods utilizing characters and sub-characters information, such as character-based CWE and sub-character-based JWE, only learn embedding of characters and sub-characters by predicting the target word. We believe it's not enough to improve the quality of characters and sub-characters embedding, so we propose a method called HJWE which predicts the target word, characters and sub-characters in the target word at the same time. The experiment result shows that our method performs best on the word similarity, word analogy and text classification tasks.
Learning Chinese Word Embeddings from Stroke, Structure and Pinyin of Characters

Yun Zhang,Yongguo Liu,Jiajing Zhu,Ziqiang Zheng,Xiaofeng Liu,Weiguang Wang,Zijie Chen,Shuangqing Zhai

DOI: https://doi.org/10.1145/3357384.3358005

2019-11-03

Abstract:Chinese word embeddings have recently attracted much attention in natural language processing (NLP). Existing researches learn Chinese word embeddings based on characters, radicals, components and stroke n-gram. Besides abovementioned features, Chinese characters also own structure and pinyin features. In this paper, we design feature substring, a super set of radicals, components and stroke n-gram with structure and pinyin information, to integrate stroke, structure and pinyin features of Chinese characters and capture the semantics of Chinese words. Based on the feature substring, we propose a novel method ssp2vec to predict the contextual words based on the feature substrings of the target words for learning Chinese word embeddings. It is based on our observation that exploiting the morphological information (stroke and structure) and the phonetic information (pinyin) is crucial for capturing the meanings of Chinese words. Meanwhile, the phonetic information (pinyin) can assist the model to distinguish Chinese words. Experimental results on word analogy, word similarity, text classification and named entity recognition tasks show that the proposed method obtains better results than state-of-the-art approaches.
VCWE: Visual Character-Enhanced Word Embeddings

Chi Sun,Xipeng Qiu,Xuanjing Huang

DOI: https://doi.org/10.48550/arXiv.1902.08795

2019-02-23

Computation and Language

Abstract:Chinese is a logographic writing system, and the shape of Chinese characters contain rich syntactic and semantic information. In this paper, we propose a model to learn Chinese word embeddings via three-level composition: (1) a convolutional neural network to extract the intra-character compositionality from the visual shape of a character; (2) a recurrent neural network with self-attention to compose character representation into word embeddings; (3) the Skip-Gram framework to capture non-compositionality directly from the contextual information. Evaluations demonstrate the superior performance of our model on four tasks: word similarity, sentiment analysis, named entity recognition and part-of-speech tagging.
An Exploration Of Semantic Relations In Neural Word Embeddings Using Extrinsic Knowledge

Zhiwei Chen,Zhe He,Xiuwen Liu,Jiang Bian

DOI: https://doi.org/10.1109/BIBM.2017.8217836

2017-01-01

Abstract:In the recent few years, neural-network-based word embeddings have been widely used in text mining. However, the dense representations of word embeddings act as a black box and lack interpretability. Even though word embeddings are able to capture semantic regularities in free text documents, it is not clear what kinds of semantic relations can be represented by word embeddings and how semantically-related terms can be retrieved from word embeddings. In this study, we propose a novel approach to explore the semantic relations in neural embeddings using extrinsic knowledge from WordNet and Unified Medical Language System (UMLS). We trained multiple word embeddings using health-related articles in Wikipedia. We then evaluated the performance of the different word embeddings in semantic relation term retrieval tasks. This study shows that word embeddings can group terms with diverse semantic relations together.
Improved Learning of Chinese Word Embeddings with Semantic Knowledge.

Liner Yang,Maosong Sun

DOI: https://doi.org/10.1007/978-3-319-25816-4_2

2015-01-01

Abstract:While previous studies show that modeling the minimum meaning-bearing units (characters or morphemes) benefits learning vector representations of words, they ignore the semantic dependencies across these units when deriving word vectors. In this work, we propose to improve the learning of Chinese word embeddings by exploiting semantic knowledge. The basic idea is to take the semantic knowledge about words and their component characters into account when designing composition functions. Experiments show that our approach outperforms two strong baselines on word similarity, word analogy, and document classification tasks.
Field Embedding: A Unified Grain-Based Framework for Word Representation

Junjie Luo,Xi Chen,Jichao Sun,Yuejia Xiang,Ningyu Zhang,Xiang Wan

DOI: https://doi.org/10.18653/v1/2021.naacl-main.140

2021-01-01

Abstract:Word representations empowered with additional linguistic information have been widely studied and proved to outperform traditional embeddings. Current methods mainly focus on learning embeddings for words while embeddings of linguistic information (referred to as grain embeddings) are discarded after the learning. This work proposes a framework field embedding to jointly learn both word and grain embeddings by incorporating morphological, phonetic, and syntactical linguistic fields. The framework leverages an innovative fine-grained pipeline that integrates multiple linguistic fields and produces high-quality grain sequences for learning supreme word representations. A novel algorithm is also designed to learn embeddings for words and grains by capturing information that is contained within each field and that is shared across them. Experimental results of lexical tasks and downstream natural language processing tasks illustrate that our framework can learn better word embeddings and grain embeddings. Qualitative evaluations show grain embeddings effectively capture the semantic information.
Enhancing Chinese Intent Classification by Dynamically Integrating Character Features into Word Embeddings with Ensemble Techniques

Ruixi Lin,Charles Costello,Charles Jankowski

DOI: https://doi.org/10.48550/arXiv.1805.08914

2018-05-23

Computation and Language

Abstract:Intent classification has been widely researched on English data with deep learning approaches that are based on neural networks and word embeddings. The challenge for Chinese intent classification stems from the fact that, unlike English where most words are made up of 26 phonologic alphabet letters, Chinese is logographic, where a Chinese character is a more basic semantic unit that can be informative and its meaning does not vary too much in contexts. Chinese word embeddings alone can be inadequate for representing words, and pre-trained embeddings can suffer from not aligning well with the task at hand. To account for the inadequacy and leverage Chinese character information, we propose a low-effort and generic way to dynamically integrate character embedding based feature maps with word embedding based inputs, whose resulting word-character embeddings are stacked with a contextual information extraction module to further incorporate context information for predictions. On top of the proposed model, we employ an ensemble method to combine single models and obtain the final result. The approach is data-independent without relying on external sources like pre-trained word embeddings. The proposed model outperforms baseline models and existing methods.
Chinese Computational Linguistics and Natural Language Processing Based on Naturally Annotated Big Data - 17th China National Conference, CCL 2018, and 6th International Symposium, NLP-NABD 2018, Changsha, China, October 19-21, 2018, Proceedings.

Maosong Sun,Xiaojie Wang,Zhiyuan Liu,Yang Liu,Ting Liu

DOI: https://doi.org/10.1007/978-3-030-01716-3

2018-01-01

Abstract:The conventional Chinese word embedding model is similar to the English word embedding model in modeling text, simply uses the Chinese word or character as the minimum processing unit of the text, without using the semantic information about Chinese characters and the radicals in Chinese words. To this end, we proposed a radical enhanced Chinese word embedding in this paper. The model uses conversion and radical escaping mechanisms to extract the intrinsic information in Chinese corpus. Through the improved parallel dual-channel network model on a CBOW-like model, the word information context is used together with the Chinese character radical information context to predict the target word. Therefore, the word vector generated by the model can fully reflect the semantic information contained in the radicals. Compared with other similar models by word analogy and similarity experiments, the results showed that our model has effectively improved the accuracy of word vector expression and the direct relevance of similar words.
Learning Word Embeddings from Intrinsic and Extrinsic Views

Jifan Chen,Kan Chen,Xipeng Qiu,Qi Zhang,Xuanjing Huang,Zheng Zhang

2016-01-01

Abstract:While word embeddings are currently predominant for natural language processing, most of existing models learn them solely from their contexts. However, these context-based word embeddings are limited since not all words' meaning can be learned based on only context. Moreover, it is also difficult to learn the representation of the rare words due to data sparsity problem. In this work, we address these issues by learning the representations of words by integrating their intrinsic (descriptive) and extrinsic (contextual) information. To prove the effectiveness of our model, we evaluate it on four tasks, including word similarity, reverse dictionaries,Wiki link prediction, and document classification. Experiment results show that our model is powerful in both word and document modeling.
Radical and Stroke-Enhanced Chinese Word Embeddings Based on Neural Networks

Shirui Wang,Wenan Zhou,Qiang Zhou

DOI: https://doi.org/10.1007/s11063-020-10289-6

IF: 2.565

2020-07-04

Neural Processing Letters

Abstract:The internal structural information of words has proven to be very effective for learning Chinese word embeddings. However, most previous attempts made a single form extraction of internal feature to learn representations, ignoring the comprehensive combination of such information. And they focused only on explicit feature of internal structures, even though these structures still have the implicit semantics of words. In this paper, we propose Radical and Stroke-enhanced Word Embeddings (RSWE), a novel method based on neural networks for learning Chinese word embeddings with joint guidance from semantic and morphological internal information. RSWE enables an embedding model to learn simultaneously from (1) implicit semantic information that is exploited from the radicals, and (2) stroke n-grams information that can be explicitly obtained from Chinese words. In the learning process, RSWE uses stroke n-grams to capture the local structural feature of words, and integrates the implicit information exploited from radicals to enhance the semantic of embeddings. Through this combination procedure, semantics of Chinese words are effectively transferred into the learned embeddings. We evaluate the effectiveness of RSWE on word similarity computation, word analogy reasoning, performance over dimensions, performance over learning corpus size, and named entity recognition tasks, the experimental results show that our model outperforms existing state-of-the-art approaches.

computer science, artificial intelligence
Multiple Character Embeddings for Chinese Word Segmentation

Jingkang Wang,Jianing Zhou,Jie Zhou,Gongshen Liu

DOI: https://doi.org/10.48550/arXiv.1808.04963

2019-05-30

Abstract:Chinese word segmentation (CWS) is often regarded as a character-based sequence labeling task in most current works which have achieved great success with the help of powerful neural networks. However, these works neglect an important clue: Chinese characters incorporate both semantic and phonetic meanings. In this paper, we introduce multiple character embeddings including Pinyin Romanization and Wubi Input, both of which are easily accessible and effective in depicting semantics of characters. We propose a novel shared Bi-LSTM-CRF model to fuse linguistic features efficiently by sharing the LSTM network during the training procedure. Extensive experiments on five corpora show that extra embeddings help obtain a significant improvement in labeling accuracy. Specifically, we achieve the state-of-the-art performance in AS and CityU corpora with F1 scores of 96.9 and 97.3, respectively without leveraging any external lexical resources.

Computation and Language
Learning Sense-specific Word Embeddings By Exploiting Bilingual Resources.

Jiang Guo,Wanxiang Che,Haifeng Wang,Ting Liu

2014-01-01

Abstract:Recent work has shown success in learning word embeddings with neural network language models (NNLM). However, the majority of previous NNLMs represent each word with a single embedding, which fails to capture polysemy. In this paper, we address this problem by representing words with multiple and sense-specific embeddings, which are learned from bilingual parallel data. We evaluate our embeddings using the word similarity measurement and show that our approach is significantly better in capturing the sense-level word similarities. We further feed our embeddings as features in Chinese named entity recognition and obtain noticeable improvements against single embeddings.
An Improved Historical Embedding without Alignment

Xiaofei Xu,Ke Deng,Fei Hu,Li Li

DOI: https://doi.org/10.48550/arXiv.1910.08692

2019-10-19

Computation and Language

Abstract:Many words have evolved in meaning as a result of cultural and social change. Understanding such changes is crucial for modelling language and cultural evolution. Low-dimensional embedding methods have shown promise in detecting words' meaning change by encoding them into dense vectors. However, when exploring semantic change of words over time, these methods require the alignment of word embeddings across different time periods. This process is computationally expensive, prohibitively time consuming and suffering from contextual variability. In this paper, we propose a new and scalable method for encoding words from different time periods into one dense vector space. This can greatly improve performance when it comes to identifying words that have changed in meaning over time. We evaluated our method on dataset from Google Books N-gram. Our method outperformed three other popular methods in terms of the number of words correctly identified to have changed in meaning. Additionally, we provide an intuitive visualization of the semantic evolution of some words extracted by our method
Exploiting Word Semantics to Enrich Character Representations of Chinese Pre-trained Models

Wenbiao Li,Rui Sun,Yunfang Wu

DOI: https://doi.org/10.1007/978-3-031-17120-8_1

2022-01-01

Abstract:Most of the Chinese pre-trained models adopt characters as basic units for downstream tasks. However, these models ignore the information carried by words and thus lead to the loss of some important semantics. In this paper, we propose a new method to exploit word structure and integrate lexical semantics into character representations of pre-trained models. Specifically, we project a word’s embedding into its internal characters’ embeddings according to the similarity weight. To strengthen the word boundary information, we mix the representations of the internal characters within a word. After that, we apply a word-to-character alignment attention mechanism to emphasize important characters by masking unimportant ones. Moreover, in order to reduce the error propagation caused by word segmentation, we present an ensemble approach to combine segmentation results given by different tokenizers. The experimental results show that our approach achieves superior performance over the basic pre-trained models BERT, BERT-wwm and ERNIE on different Chinese NLP tasks: sentiment classification, sentence pair matching, natural language inference and machine reading comprehension. We make further analysis to prove the effectiveness of each component of our model.
Learning Chinese-Japanese Bilingual Word Embedding by Using Common Characters.

Wang Jilei,Luo Shiying,Li Yanning,Xia Shu-Tao

DOI: https://doi.org/10.1007/978-3-319-47650-6_7

2016-01-01

Abstract:Bilingual word embedding, which maps word embedding of two languages into one vector space, has been widely applied in the domain of machine translation, word sense disambiguation and so on. However, no model has been universally accepted for learning bilingual word embedding. In this work, we propose a novel model named CJ-BOC to learn Chinese-Japanese word embeddings. Given Chinese and Japanese share a large portion of common characters, we exploit them in our training process. We demonstrated the effectiveness of such exploitation through theoretical and also experimental study. To evaluate the performance of CJ-BOC, we conducted a comprehensive experiment, which reveals its speed advantage, and high quality of acquired word embeddings as well.

Improve Chinese Word Embeddings by Exploiting Internal Structure.

Improve Word Embedding Using Both Writing and Pronunciation.

Enhanced Double-Carrier Word Embedding Via Phonetics and Writing

Using Context-to-Vector with Graph Retrofitting to Improve Word Embeddings

Joint Learning of Character and Word Embeddings.

Pronunciation-Enhanced Chinese Word Embedding

Hierarchical Joint Learning for Chinese Word Embeddings

Learning Chinese Word Embeddings from Stroke, Structure and Pinyin of Characters

VCWE: Visual Character-Enhanced Word Embeddings

An Exploration Of Semantic Relations In Neural Word Embeddings Using Extrinsic Knowledge

Improved Learning of Chinese Word Embeddings with Semantic Knowledge.

Field Embedding: A Unified Grain-Based Framework for Word Representation

Enhancing Chinese Intent Classification by Dynamically Integrating Character Features into Word Embeddings with Ensemble Techniques

Chinese Computational Linguistics and Natural Language Processing Based on Naturally Annotated Big Data - 17th China National Conference, CCL 2018, and 6th International Symposium, NLP-NABD 2018, Changsha, China, October 19-21, 2018, Proceedings.

Learning Word Embeddings from Intrinsic and Extrinsic Views

Radical and Stroke-Enhanced Chinese Word Embeddings Based on Neural Networks

Multiple Character Embeddings for Chinese Word Segmentation

Learning Sense-specific Word Embeddings By Exploiting Bilingual Resources.

An Improved Historical Embedding without Alignment

Exploiting Word Semantics to Enrich Character Representations of Chinese Pre-trained Models

Learning Chinese-Japanese Bilingual Word Embedding by Using Common Characters.