Abstract:Unstructured text provides decision-makers with a rich data source in many domains, ranging from product reviews in retail to nursing notes in healthcare. To leverage this information, words are typically translated into word embeddings---vectors that encode the semantic relationships between words---through unsupervised learning algorithms such as matrix factorization. However, learning word embeddings from new domains with limited training data can be challenging, because the meaning/usage may be different in the new domain, e.g., the word ``positive'' typically has positive sentiment, but often has negative sentiment in medical notes since it may imply that a patient tested positive for a disease. In practice, we expect that only a small number of domain-specific words may have new meanings. We propose an intuitive two-stage estimator that exploits this structure via a group-sparse penalty to efficiently transfer learn domain-specific word embeddings by combining large-scale text corpora (such as Wikipedia) with limited domain-specific text data. We bound the generalization error of our transfer learning estimator, proving that it can achieve high accuracy with substantially less domain-specific data when only a small number of embeddings are altered between domains. Furthermore, we prove that all local minima identified by our nonconvex objective function are statistically indistinguishable from the global minimum under standard regularization conditions, implying that our estimator can be computed efficiently. Our results provide the first bounds on group-sparse matrix factorization, which may be of independent interest. We empirically evaluate our approach compared to state-of-the-art fine-tuning heuristics from natural language processing.

Learning Cross-lingual Word Embeddings Via Matrix Co-factorization.

Learning Visually Aligned Semantic Graph for Cross-Modal Manifold Matching.

Exploiting Common Characters in Chinese and Japanese to Learn Cross-Lingual Word Embeddings Via Matrix Factorization.

Joint Learning of Character and Word Embeddings.

Learning Tibetan-Chinese Cross-Lingual Word Embeddings

Improving Vector Space Word Representations Via Kernel Canonical Correlation Analysis

Collaboratively Improving Topic Discovery and Word Embeddings by Coordinating Global and Local Contexts

Group-Sparse Matrix Factorization for Transfer Learning of Word Embeddings

Unsupervised Learning of Multi-Sense Embedding with Matrix Factorization and Sparse Soft Clustering

MTLAN: Multi-Task Learning and Auxiliary Network for Enhanced Sentence Embedding

Cluster-driven Model for Improved Word and Text Embedding

A Cross-Lingual Sentence Pair Interaction Feature Capture Model Based on Pseudo-Corpus and Multilingual Embedding

Unsupervised Cross-Lingual Sentence Representation Learning via Linguistic Isomorphism

Word Embedding Revisited: A New Representation Learning and Explicit Matrix Factorization Perspective.

Online Learning of Interpretable Word Embeddings

Cross-Modality Retrieval by Joint Correlation Learning

Learning Multilingual Sentence Embeddings From Monolingual Corpus

Jointly Learning Bilingual Word Embeddings and Alignments

Supervised Matrix Factorization for Cross-Modality Hashing.

Incorporating Subword Information into Matrix Factorization Word Embeddings

Learnable Multi-view Matrix Factorization with Graph Embedding and Flexible Loss