Abstract:In this paper we perform a rigorous mathematical analysis of the word2vec model, especially when it is equipped with the Skip-gram learning scheme. Our goal is to explain how embeddings, that are now widely used in NLP (Natural Language Processing), are influenced by the distribution of terms in the documents of the considered corpus. We use a mathematical formulation to shed light on how the decision to use such a model makes implicit assumptions on the structure of the language. We show how Markovian assumptions, that we discuss, lead to a very clear theoretical understanding of the formation of embeddings, and in particular the way it captures what we call frequentist synonyms. These assumptions allow to produce generative models and to conduct an explicit analysis of the loss function commonly used by these NLP techniques. Moreover, we produce synthetic corpora with different levels of structure and show empirically how the word2vec algorithm succeed, or not, to learn them. It leads us to empirically assess the capability of such models to capture structures on a corpus of around 42 millions of financial News covering 12 years. That for, we rely on the Loughran-McDonald Sentiment Word Lists largely used on financial texts and we show that embeddings are exposed to mixing terms with opposite polarity, because of the way they can treat antonyms as frequentist synonyms. Beside we study the non-stationarity of such a financial corpus, that has surprisingly not be documented in the literature. We do it via time series of cosine similarity between groups of polarized words or company names, and show that embedding are indeed capturing a mix of English semantics and joined distribution of words that is difficult to disentangle.

Early in the morning Mr . Sider , an estate lawyer , pores over last wills and testaments

Discourse Relations Detection Via a Mixed Generative-Discriminative Framework

Enhanced Double-Carrier Word Embedding Via Phonetics and Writing

Using Context-to-Vector with Graph Retrofitting to Improve Word Embeddings

Field Embedding: A Unified Grain-Based Framework for Word Representation

Learning Word Embeddings from Intrinsic and Extrinsic Views

An Exploration Of Semantic Relations In Neural Word Embeddings Using Extrinsic Knowledge

Integrating Word Embeddings and Traditional NLP Features to Measure Textual Entailment and Semantic Relatedness of Sentence Pairs

Discourse Relation Embeddings: Representing the Relations between Discourse Segments in Social Media

Word Embedding Based Document Similarity for the Inferring of Penalty.

Learning Sentiment-Inherent Word Embedding for Word-Level and Sentence-Level Sentiment Analysis

Do Word Embeddings Really Understand Loughran-McDonald's Polarities?

Utility of General and Specific Word Embeddings for Classifying Translational Stages of Research

Word Embeddings Are Steers for Language Models

Learning Word Sense Embeddings from Word Sense Definitions

Analyzing Structures in the Semantic Vector Space: A Framework for Decomposing Word Embeddings

Using a Chinese Lexicon to Learn Sense Embeddings and Measure Semantic Similarity.

Learning Chinese Word Embeddings from Stroke, Structure and Pinyin of Characters

Conceptualized and Contextualized Gaussian Embedding.

Learning Word Embedding with Better Distance Weighting and Window Size Scheduling

Word Embedding Dimension Reduction via Weakly-Supervised Feature Selection