Abstract:In this paper we perform a rigorous mathematical analysis of the word2vec model, especially when it is equipped with the Skip-gram learning scheme. Our goal is to explain how embeddings, that are now widely used in NLP (Natural Language Processing), are influenced by the distribution of terms in the documents of the considered corpus. We use a mathematical formulation to shed light on how the decision to use such a model makes implicit assumptions on the structure of the language. We show how Markovian assumptions, that we discuss, lead to a very clear theoretical understanding of the formation of embeddings, and in particular the way it captures what we call frequentist synonyms. These assumptions allow to produce generative models and to conduct an explicit analysis of the loss function commonly used by these NLP techniques. Moreover, we produce synthetic corpora with different levels of structure and show empirically how the word2vec algorithm succeed, or not, to learn them. It leads us to empirically assess the capability of such models to capture structures on a corpus of around 42 millions of financial News covering 12 years. That for, we rely on the Loughran-McDonald Sentiment Word Lists largely used on financial texts and we show that embeddings are exposed to mixing terms with opposite polarity, because of the way they can treat antonyms as frequentist synonyms. Beside we study the non-stationarity of such a financial corpus, that has surprisingly not be documented in the literature. We do it via time series of cosine similarity between groups of polarized words or company names, and show that embedding are indeed capturing a mix of English semantics and joined distribution of words that is difficult to disentangle.

Political Footprints: Political Discourse Analysis using Pre-Trained Word Vectors

Detecting policy preferences and dynamics in the UN general debate with neural word embeddings

Political DEBATE: Efficient Zero-shot and Few-shot Classifiers for Political Text

Understanding Political Polarisation using Language Models: A dataset and method

WMDecompose: A Framework for Leveraging the Interpretable Properties of Word Mover's Distance in Sociocultural Analysis

Studying Political Decision Making With Automatic Text Analysis

MARMOT: A Deep Learning Framework for Constructing Multimodal Representations for Vision-and-Language Tasks

Leveraging Large Language Models for Fuzzy String Matching in Political Science

Dialectograms: Machine Learning Differences between Discursive Communities

Evaluating large language models for user stance detection on X (Twitter)

Words are Malleable: Computing Semantic Shifts in Political and Media Discourse

"Read My Lips": Using Automatic Text Analysis to Classify Politicians by Party and Ideology

Identification, Interpretability, and Bayesian Word Embeddings

The Use of Machine Learning Methods in Political Science: An In-Depth Literature Review

Which Identities Are Mobilized: Towards an automated detection of social group appeals in political texts

From Pretraining Data to Language Models to Downstream Tasks: Tracking the Trails of Political Biases Leading to Unfair NLP Models

No Word Embedding Model Is Perfect: Evaluating the Representation Accuracy for Social Bias in the Media

Do Word Embeddings Really Understand Loughran-McDonald's Polarities?

Analyzing differences between discursive communities using dialectograms

Positioning Political Texts with Large Language Models by Asking and Averaging

Large Language Models' Detection of Political Orientation in Newspapers