Abstract:Most popular word embedding techniques involve implicit or explicit factorization of a word co-occurrence based matrix into low rank factors. In this paper, we aim to generalize this trend by using numerical methods to factor higher-order word co-occurrence based arrays, or \textit{tensors}. We present four word embeddings using tensor factorization and analyze their advantages and disadvantages. One of our main contributions is a novel joint symmetric tensor factorization technique related to the idea of coupled tensor factorization. We show that embeddings based on tensor factorization can be used to discern the various meanings of polysemous words without being explicitly trained to do so, and motivate the intuition behind why this works in a way that doesn't with existing methods. We also modify an existing word embedding evaluation metric known as Outlier Detection [Camacho-Collados and Navigli, 2016] to evaluate the quality of the order-$N$ relations that a word embedding captures, and show that tensor-based methods outperform existing matrix-based methods at this task. Experimentally, we show that all of our word embeddings either outperform or are competitive with state-of-the-art baselines commonly used today on a variety of recent datasets. Suggested applications of tensor factorization-based word embeddings are given, and all source code and pre-trained vectors are publicly available online.

Incorporating Subword Information into Matrix Factorization Word Embeddings

Enriching Word Vectors with Subword Information

Learning Cross-lingual Word Embeddings Via Matrix Co-factorization.

Word Embedding Revisited: A New Representation Learning and Explicit Matrix Factorization Perspective.

Adapting Word Embeddings to New Languages with Morphological and Phonological Subword Representations

Analyzing Structures in the Semantic Vector Space: A Framework for Decomposing Word Embeddings

Evaluating Subword Tokenization: Alien Subword Composition and OOV Generalization Challenge

A Generative Word Embedding Model and its Low Rank Positive Semidefinite Solution

Effective Subword Segmentation for Text Comprehension

A Joint Model for Word Embedding and Word Morphology

Co-learning of Word Representations and Morpheme Representations.

Learning Effective Word Embedding Using Morphological Word Similarity

Beyond Bilingual: Multi-sense Word Embeddings using Multilingual Context

A Systematic Analysis of Subwords and Cross-Lingual Transfer in Multilingual Translation

BioWordVec, improving biomedical word embeddings with subword information and MeSH

Tomato, Tomahto, Tomate: Measuring the Role of Shared Semantics among Subwords in Multilingual Language Models

Investigating Language Universal and Specific Properties in Word Embeddings

Word Embeddings via Tensor Factorization

On the Effect of (Near) Duplicate Subwords in Language Modelling

Embedding Word Similarity with Neural Machine Translation

Improved Learning of Chinese Word Embeddings with Semantic Knowledge.