Abstract:It is widely accepted that traditional word embedding models, which rely on distributional semantics hypothesis, are relatively limited for contrast meaning problem. Distributional semantics hypothesis indicates that words lying in similar contexts have similar representations in vector space. Nevertheless, synonyms and antonyms often locate in similar contexts, which means they appear close to each other in vector space. Hence, it is of great difficulty to distinguish antonyms from synonyms. To address this challenge, we propose an optimization model, named Lexicon-based Word Embedding Tuning (LWET) model. The goal of LWET is to incorporate reliable semantic lexicons to tune the distributions of pre-trained word embeddings in the vector space so as to improve their ability of distinguishing antonyms from synonyms. To speed up the training process of LWET, we propose two approximation algorithms, including positive sampling and quasi-hierarchical softmax. Compared with quasi-hierarchical softmax, positive sampling is faster, however, at the cost of worse performance. In experiments, LWET and other state-of-the-art models are tested on antonyms recognition, distinguishing antonyms from synonyms and word similarity. The results of the first two experiments show that LWET significantly improves the ability of word embeddings to detect antonyms, thus achieving the state-of-the-art performance. On word similarity, LWET gets slightly better performance than the state-of-the-art models. It means that LWET can remain and strengthen the semantic structure rather than destroy it when tuning word distributions in vector space. In general, compared with related work, LWET can not only achieve similar or even better performance, but also speed up the training process.

A Mixture-of-Experts Model for Antonym-Synonym Discrimination

Modeling Task Relationships in Multivariate Soft Sensor with Balanced Mixture-of-Experts

A Mixture Model for Expert Finding

An Experimental Study on Discriminating Chinese Near Synonyms: Contrasts Between Machine Learning Systems and Second Language Learners

HMoE: Heterogeneous Mixture of Experts for Language Modeling

Revisit Word Embeddings with Semantic Lexicons for Modeling Lexical Contrast

Diversifying the Expert Knowledge for Task-Agnostic Pruning in Sparse Mixture-of-Experts

A Survey on Mixture of Experts

Not Eliminate but Aggregate: Post-Hoc Control over Mixture-of-Experts to Address Shortcut Shifts in Natural Language Understanding

MoDE: A Mixture-of-Experts Model with Mutual Distillation among the Experts

Improving Word Embeddings for Antonym Detection Using Thesauri and SentiWordNet.

Multi-Head Mixture-of-Experts

Explainable data-driven modeling via mixture of experts: towards effective blending of grey and black-box models

Learning a Mixture of Granularity-Specific Experts for Fine-Grained Categorization

Not All Synonyms Are Created Equal: Incorporating Similarity of Synonyms to Enhance Word Embeddings

Prediction Sets for High-Dimensional Mixture of Experts Models

HyperMoE: Towards Better Mixture of Experts via Transferring Among Experts

Mix of Experts Language Model for Named Entity Recognition

Generalization Error Analysis for Sparse Mixture-of-Experts: A Preliminary Study

Expert-Token Resonance: Redefining MoE Routing through Affinity-Driven Active Selection

Learning Mixtures of Experts with EM