Abstract:Most of the modern natural language processing (NLP) techniques are based on the vector space models of language, in which each word is represented by a vector in a high dimensional space. One of the earliest successes was demonstrated by the four-term analogical reasoning task: what is to C as B is to A? The trained word vectors form "parallelograms" representing the quadruple of words in analogy. This discovery in NLP offers us insight into our understanding of human semantic representation of words via analogical reasoning. Despite successful applications of the large-scale language models, it has not been fully understood why such parallelograms emerge by learning through natural language data. As the vector space model is not optimized to form parallelograms, the key structure related to geometric shapes of word vectors is expected to be in the data, rather than the models. In the present article, we test our hypothesis that such parallelogram arrangement of word vectors readily exists in the co-occurrence statistics of language. Our approach focuses more on the data itself, and it is different from the existing theoretical approach trying to find the mechanism of parallelogram formation in the algorithms and/or vector arithmetic operations on word vectors. First, our analysis suggested that analogical reasoning is possible by decomposition of the bigram co-occurrence matrix. Second, we demonstrated the formation of a parallelepiped, a more structured geometric object than a parallelogram, by creating a small artificial corpus and its word vectors. With these results, we propose a refined form of distributional hypothesis pointing out an isomorphism between a sort of symmetry or exchangeability and word co-occurrence statistics.

Co-occurrence Vectors from Corpora vs. Distance Vectors from Dictionaries

Contextualized Word Vector-based Methods for Discovering Semantic Differences with No Training nor Word Alignment

Corpora Compared: The Case of the Swedish Gigaword & Wikipedia Corpora

Distributional hypothesis as isomorphism between word-word co-occurrence and analogical parallelograms

Characterizing how 'distributional' NLP corpora distance metrics are

Coarse-Grained Sense Inventories Based on Semantic Matching between English Dictionaries

Why PairDiff works? -- A Mathematical Analysis of Bilinear Relational Compositional Operators for Analogy Detection

Semantic Similarity Based on Corpus Statistics and Lexical Taxonomy

Improving Word Vector with Prior Knowledge in Semantic Dictionary.

Syntactic Complexity of Different Text Types: from the Perspective of Dependency Distance Both Linearly and Hierarchically

Distributional Measures of Semantic Distance: A Survey

Take and Took, Gaggle and Goose, Book and Read: Evaluating the Utility of Vector Differences for Lexical Relation Learning

Dynamic Valency And Dependency Distance

Measuring Word Significance using Distributed Representations of Words

Correlation Coefficients and Semantic Textual Similarity

Assessing the contribution of shallow and deep knowledge sources for word sense disambiguation

A space for the quantum world

Construction of a Japanese Word Similarity Dataset

Improving Vector Space Word Representations Via Kernel Canonical Correlation Analysis

Simple, Interpretable and Stable Method for Detecting Words with Usage Change across Corpora

A comparative study for wordnet guided text representation