Abstract:Word sense disambiguation is a crucial task to test whether a model can perform deep understanding. Nowadays, the rise of pre-trained language models facilitates substantial success in such tasks. However, most current pre-training tasks are taken in place of the token level while ignoring the linguistic sense of the tokens themselves. Thus it is questionable whether the token-predicting objectives are enough to learn the polysemy and disambiguate senses. To explore this question, we introduce RS-BERT, a radical enhanced sense embedding model with a novel pre-training objective, sense-aware language modeling, which introduces additional sense-level information to the model. For each training step, we first predict the senses and then update the model given the predicted senses. During training, we alternately perform the above two steps in an expectation–maximization manner. Besides, we also introduce radical information to RS-BERT at the beginning of pre-training. We conduct experiments on two Chinese word sense disambiguation datasets. Experimental results show that RS-BERT is competitive. When combined with other dedicated adaptions for specific datasets, RS-BERT shows impressive performance. Moreover, our analysis shows that RS-BERT successfully clusters Chinese characters into various senses. The experiment results demonstrate that the token-predicting objectives are not enough and the sense-level objective performs better for polysemy and sense disambiguation.

Unsupervised Translation Disambiguation Based on Equivalent PseudoTranslation Model

Unsupervised Word Sense Disambiguation Based on WordNet

An Equivalent Pseudoword Solution to Chinese Word Sense Disambiguation

Comparison of Web-Based Unsupervised Translation Disambiguation Word Model and N-gram Model

Unsupervised Translation Disambiguation Based on Web Indirect Association of Bilingual Word

Unsupervised translation disambiguation based on maximum web bilingual relatedness: web as lexicon

Application of the transformer model algorithm in chinese word sense disambiguation: a case study in chinese language

Research on dual pattern of unsupervised and supervised word sense disambiguation

An Approach to Extract Named Entity Translingual Equivalence

Unsupervised translation disambiguation based on mining Web relatedness of bilingual words

Chinese word sense disambiguation using hownet

A Comparison of Word Embeddings for English and Cross-Lingual Chinese Word Sense Disambiguation

RS-BERT: Pre-training radical enhanced sense embedding for Chinese word sense disambiguation

Learning Sense Representation from Word Representation for Unsupervised Word Sense Disambiguation (student Abstract).

Integrating Weakly Supervised Word Sense Disambiguation into Neural Machine Translation

Minimum Normalized Google Distance for Unsupervised Multilingual Chinese-English Word Sense Disambiguation

Constructing High Quality Sense-specific Corpus and Word Embedding Via Unsupervised Elimination of Pseudo Multi-sense.

Try to Substitute: an Unsupervised Chinese Word Sense Disambiguation Method Based on HowNet.

Unsupervised Word Sense Discrimination Improves Construction of the Wordnets.

Integrating Multiple Knowledge Sources to Disambiguate Word Sense: An Exemplar-Based Approach

Semi Supervised Preposition-Sense Disambiguation using Multilingual Data