Abstract:The words of a language reflect the structure of the human mind, allowing us to transmit thoughts between individuals. However, language can represent only a subset of our rich and detailed cognitive architecture. Here, we ask what kinds of common knowledge (semantic memory) are captured by word meanings (lexical semantics). We examine a prominent computational model that represents words as vectors in a multidimensional space, such that proximity between word-vectors approximates semantic relatedness. Because related words appear in similar contexts, such spaces - called "word embeddings" - can be learned from patterns of lexical co-occurrences in natural language. Despite their popularity, a fundamental concern about word embeddings is that they appear to be semantically "rigid": inter-word proximity captures only overall similarity, yet human judgments about object similarities are highly context-dependent and involve multiple, distinct semantic features. For example, dolphins and alligators appear similar in size, but differ in intelligence and aggressiveness. Could such context-dependent relationships be recovered from word embeddings? To address this issue, we introduce a powerful, domain-general solution: "semantic projection" of word-vectors onto lines that represent various object features, like size (the line extending from the word "small" to "big"), intelligence (from "dumb" to "smart"), or danger (from "safe" to "dangerous"). This method, which is intuitively analogous to placing objects "on a mental scale" between two extremes, recovers human judgments across a range of object categories and properties. We thus show that word embeddings inherit a wealth of common knowledge from word co-occurrence statistics and can be flexibly manipulated to express context-dependent meanings.

The Interplay of Semantics and Morphology in Word Embeddings

Do Multi-Sense Embeddings Improve Natural Language Understanding?

Joint Semantic Synthesis and Morphological Analysis of the Derived Word

Investigating Language Universal and Specific Properties in Word Embeddings

Semantic projection: recovering human knowledge of multiple, distinct object features from word embeddings

Visual Exploration and Comparison of Word Embeddings.

A Joint Model for Word Embedding and Word Morphology

Semantic Word Cloud Generation Based on Word Embeddings

Learning Effective Word Embedding Using Morphological Word Similarity

Enriching Word Embeddings with Temporal and Spatial Information

Do Word Embeddings Really Understand Loughran-McDonald's Polarities?

Semantic properties of English nominal pluralization: Insights from word embeddings

Improving Distributional Similarity with Lessons Learned from Word Embeddings

Contextualized Word Embeddings Encode Aspects of Human-Like Word Sense Knowledge

Tackling Polysemanticity with Neuron Embeddings

Morphological Inflection with Phonological Features

Tomato, Tomahto, Tomate: Measuring the Role of Shared Semantics among Subwords in Multilingual Language Models

Lexical semantics enhanced neural word embeddings

Assessing the Role of Lexical Semantics in Cross-lingual Transfer through Controlled Manipulations

Ontological Relations from Word Embeddings

Unsupervised Morphological Expansion of Small Datasets for Improving Word Embeddings