Abstract:The development of state-of-the-art (SOTA) Natural Language Processing (NLP) systems has steadily been establishing new techniques to absorb the statistics of linguistic data. These techniques often trace well-known constructs from traditional theories, and we study these connections to close gaps around key NLP methods as a means to orient future work. For this, we introduce an analytic model of the statistics learned by seminal algorithms (including GloVe and Word2Vec), and derive insights for systems that use these algorithms and the statistics of co-occurrence, in general. In this work, we derive -- to the best of our knowledge -- the first known solution to Word2Vec's softmax-optimized, skip-gram algorithm. This result presents exciting potential for future development as a direct solution to a deep learning (DL) language model's (LM's) matrix factorization. However, we use the solution to demonstrate a seemingly-universal existence of a property that word vectors exhibit and which allows for the prophylactic discernment of biases in data -- prior to their absorption by DL models. To qualify our work, we conduct an analysis of independence, i.e., on the density of statistical dependencies in co-occurrence models, which in turn renders insights on the distributional hypothesis' partial fulfillment by co-occurrence statistics.

Linking GloVe with Word2vec

Linking GloVe with word2vec.

Using Context-to-Vector with Graph Retrofitting to Improve Word Embeddings

Boosting Collective Entity Linking via Type-Guided Semantic Embedding.

Embedding Words in Non-Vector Space with Unsupervised Graph Learning

WOVe: Incorporating Word Order in GloVe Word Embeddings

Exploration on Grounded Word Embedding: Matching Words and Images with Image-Enhanced Skip-Gram Model

SemGloVe: Semantic Co-occurrences for GloVe from BERT

Post-Processing of Word Representations via Variance Normalization and Dynamic Embedding

Language Models with Pre-Trained (GloVe) Word Embeddings

To Know by the Company Words Keep and What Else Lies in the Vicinity

Learning representations for gene ontology terms by jointly encoding graph structure and textual node descriptors

Visual Exploration and Comparison of Word Embeddings.

Word Embedding Revisited: A New Representation Learning and Explicit Matrix Factorization Perspective.

Vec2Gloss: definition modeling leveraging contextualized vectors with Wordnet gloss

Embedded Representation of Relation Words with Visual Supervision

Learning Tibetan-Chinese Cross-Lingual Word Embeddings

cw2vec: Learning Chinese Word Embeddings with Stroke n-gram Information

Statistical Uncertainty in Word Embeddings: GloVe-V

Gram2Vec: An Interpretable Document Vectorizer

Visual Word2Vec (vis-w2v): Learning Visually Grounded Word Embeddings Using Abstract Scenes