Abstract:Word embedding algorithms produce very reliable feature representations of words that are used by neural network models across a constantly growing multitude of NLP tasks. As such, it is imperative for NLP practitioners to understand how their word representations are produced, and why they are so impactful. The present work presents the Simple Embedder framework, generalizing the state-of-the-art existing word embedding algorithms (including Word2vec (SGNS) and GloVe) under the umbrella of generalized low rank models. We derive that both of these algorithms attempt to produce embedding inner products that approximate pointwise mutual information (PMI) statistics in the corpus. Once cast as Simple Embedders, comparison of these models reveals that these successful embedders all resemble a straightforward maximum likelihood estimate (MLE) of the PMI parametrized by the inner product (between embeddings). This MLE induces our proposed novel word embedding model, Hilbert-MLE, as the canonical representative of the Simple Embedder framework. We empirically compare these algorithms with evaluations on 17 different datasets. Hilbert-MLE consistently observes second-best performance on every extrinsic evaluation (news classification, sentiment analysis, POS-tagging, and supersense tagging), while the first-best model depends varying on the task. Moreover, Hilbert-MLE consistently observes the least variance in results with respect to the random initialization of the weights in bidirectional LSTMs. Our empirical results demonstrate that Hilbert-MLE is a very consistent word embedding algorithm that can be reliably integrated into existing NLP systems to obtain high-quality results.

A Simple Language Model based on PMI Matrix Approximations

PMIVec: a Word Embedding Model Guided by Point-Wise Mutual Information Criterion.

Revisiting Simple Neural Probabilistic Language Models

A Quantum Expectation Value Based Language Model with Application to Question Answering

A neural probabilistic language model

Topic-based mixture language modelling

A Simple and Efficient Probabilistic Language model for Code-Mixed Text

Language Model Evaluation Beyond Perplexity

Generalizing and Hybridizing Count-based and Neural Language Models

Combining Entropy and Matrix Nuclear Norm for Enhanced Evaluation of Language Models

A comparison of correspondence analysis with PMI-based word embedding methods

Topic Optimization Method Based on Pointwise Mutual Information

Word Embedding Algorithms as Generalized Low Rank Models and their Canonical Form

Strong and Simple Baselines for Multimodal Utterance Embeddings

Language Models Implement Simple Word2Vec-style Vector Arithmetic

A Quantum Many-body Wave Function Inspired Language Modeling Approach

Refine Bigram PLSA Model by Assigning Latent Topics Unevenly

A Geometrical Approach to Topic Model Estimation

Structural Risk Minimization Principle Based Gaussian Mixture Modeling

Dense Training, Sparse Inference: Rethinking Training of Mixture-of-Experts Language Models

Beyond the Black Box: A Statistical Model for LLM Reasoning and Inference