Abstract:Abstract This article proposes a comprehensive strategy for training, evaluating, and optimizing domain-specific word2vec-based word embeddings, using social science literature as an example. Our primary objectives are: (1) to train the embeddings utilizing a corpus of social science text, (2) to test their performance against domain-unspecific embeddings using our developed intrinsic and extrinsic evaluation strategy, and (3) to enhance their performance even further by using domain knowledge. As an integral part of this approach, we present SociRel-461 , a domain-knowledge dictionary designed for the intrinsic evaluation and subsequent refinement of social science word embeddings. Using a dataset of 100,000 full-text scientific articles in sociology, we train multiple vector space models, which we then benchmark against a larger, pre-trained general language embedding model as part of our extrinsic evaluation. Furthermore, we developed a transfer learning multi-label classification task for extrinsic evaluation. Our findings reveal that domain-specific embeddings outperform their domain-unspecific counterparts in both intrinsic and extrinsic evaluations. We also investigated the retrofitting post-processing method to enhance domain-unspecific embeddings with the domain knowledge embedded in SociRel-461 . While retrofitting does not enhance our domain-specific vector space models, it significantly improves the performance of the domain-unspecific embeddings. This highlights the potential of retrofitting for the transfer of domain knowledge to domain-unspecific embeddings. Our results emphasize the importance of utilizing domain-specific word embeddings for better performance in domain specific transfer learning tasks, as they outperform conventional embeddings trained on everyday language.

Experimenting with modeling-specific word embeddings

Advancing language models through domain knowledge integration: a comprehensive approach to training, evaluation, and optimization of social scientific neural word embeddings

MUSEDA: Multilingual Unsupervised and Supervised Embedding for Domain Adaption

An Exploration Of Semantic Relations In Neural Word Embeddings Using Extrinsic Knowledge

Deconstructing and reconstructing word embedding algorithms

Cross-lingual Models of Word Embeddings: An Empirical Comparison

Evaluating Word Embedding Models: Methods and Experimental Results

Utility of General and Specific Word Embeddings for Classifying Translational Stages of Research

DEPT: Decoupled Embeddings for Pre-training Language Models

Exploring the Value of Personalized Word Embeddings

Word-class embeddings for multiclass text classification

Topic Modeling in Embedding Spaces

Beyond Bilingual: Multi-sense Word Embeddings using Multilingual Context

Debiasing Word Embeddings Improves Multimodal Machine Translation

Mixed Membership Word Embeddings for Computational Social Science

Learning Word Embeddings from Intrinsic and Extrinsic Views

Word Embedding Algorithms as Generalized Low Rank Models and their Canonical Form

IsoVec: Controlling the Relative Isomorphism of Word Embedding Spaces

Word embeddings and recurrent neural networks based on Long-Short Term Memory nodes in supervised biomedical word sense disambiguation

An Empirical Study on Post-processing Methods for Word Embeddings

VCDM: Leveraging Variational Bi-encoding and Deep Contextualized Word Representations for Improved Definition Modeling