Distributed representations of words and phrases and their compositionality. arXiv 2013

Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg Corrado, Jeffrey Dean

2013-01-01

Abstract:

What problem does this paper attempt to address?

A Model of Coherence Based on Distributed Sentence Representation.

Jiwei Li,Eduard H. Hovy

DOI: https://doi.org/10.3115/v1/d14-1218

2014-01-01

Abstract:Coherence is what makes a multi-sentence text meaningful, both logically and syntactically.To solve the challenge of ordering a set of sentences into coherent order, existing approaches focus mostly on defining and using sophisticated features to capture the cross-sentence argumentation logic and syntactic relationships.But both argumentation semantics and crosssentence syntax (such as coreference and tense rules) are very hard to formalize.In this paper, we introduce a neural network model for the coherence task based on distributed sentence representation.The proposed approach learns a syntacticosemantic representation for sentences automatically, using either recurrent or recursive neural networks.The architecture obviated the need for feature engineering, and learns sentence representations, which are to some extent able to capture the 'rules' governing coherent sentence structure.The proposed approach outperforms existing baselines and generates the stateof-art performance in standard coherence evaluation tasks 1 .
A Gloss Composition and Context Clustering Based Distributed Word Sense Representation Model.

Tao Chen,Ruifeng Xu,Yulan He,Xuan Wang

DOI: https://doi.org/10.3390/e17096007

IF: 2.738

2015-01-01

Entropy

Abstract:In recent years, there has been an increasing interest in learning a distributed representation of word sense. Traditional context clustering based models usually require careful tuning of model parameters, and typically perform worse on infrequent word senses. This paper presents a novel approach which addresses these limitations by first initializing the word sense embeddings through learning sentence-level embeddings from WordNet glosses using a convolutional neural networks. The initialized word sense embeddings are used by a context clustering based model to generate the distributed representations of word senses. Our learned representations outperform the publicly available embeddings on half of the metrics in the word similarity task, 6 out of 13 sub tasks in the analogical reasoning task, and gives the best overall accuracy in the word sense effect classification task, which shows the effectiveness of our proposed distributed distribution learning model.
Semantic Processing Through Distributed Representation of Chinese Words

Wenhui Wu,Zhiting Xiao,Yanni Wang,Xuan Guo,Yan Peng

DOI: https://doi.org/10.1109/cac.2017.8243141

2017-01-01

Abstract:In this paper, by learning the origin of the word distributed representation, knowing the distributed representation is one of the bridges of natural language processing mapping to mathematical calculations. Through the learning distributed representation model: neural network language model, CBOW model and Skip-gram model, the advantages and disadvantages of each model are clarified. Through the experiment, we can understand the mapping relationship between distributed representation calculation and natural language processing, and also clarify the research direction of the next step, that is, the use of distributed representation to sentence, paragraph, article modeling.
"The Sum of Its Parts": Joint Learning of Word and Phrase Representations with Autoencoders

Rémi Lebret,Ronan Collobert

DOI: https://doi.org/10.48550/arXiv.1506.05703

2015-06-18

Abstract:Recently, there has been a lot of effort to represent words in continuous vector spaces. Those representations have been shown to capture both semantic and syntactic information about words. However, distributed representations of phrases remain a challenge. We introduce a novel model that jointly learns word vector representations and their summation. Word representations are learnt using the word co-occurrence statistical information. To embed sequences of words (i.e. phrases) with different sizes into a common semantic space, we propose to average word vector representations. In contrast with previous methods which reported a posteriori some compositionality aspects by simple summation, we simultaneously train words to sum, while keeping the maximum information from the original vectors. We evaluate the quality of the word representations on several classical word evaluation tasks, and we introduce a novel task to evaluate the quality of the phrase representations. While our distributed representations compete with other methods of learning word representations on word evaluations, we show that they give better performance on the phrase evaluation. Such representations of phrases could be interesting for many tasks in natural language processing.

Computation and Language
Contextual Compositionality Detection with External Knowledge Bases andWord Embeddings

Dongsheng Wang,Quichi Li,Lucas Chaves Lima,Jakob grue Simonsen,Christina Lioma

DOI: https://doi.org/10.48550/arXiv.1903.08389

2019-03-20

Abstract:When the meaning of a phrase cannot be inferred from the individual meanings of its words (e.g., hot dog), that phrase is said to be non-compositional. Automatic compositionality detection in multi-word phrases is critical in any application of semantic processing, such as search engines; failing to detect non-compositional phrases can hurt system effectiveness notably. Existing research treats phrases as either compositional or non-compositional in a deterministic manner. In this paper, we operationalize the viewpoint that compositionality is contextual rather than deterministic, i.e., that whether a phrase is compositional or non-compositional depends on its context. For example, the phrase `green card' is compositional when referring to a green colored card, whereas it is non-compositional when meaning permanent residence authorization. We address the challenge of detecting this type of contextual compositionality as follows: given a multi-word phrase, we enrich the word embedding representing its semantics with evidence about its global context (terms it often collocates with) as well as its local context (narratives where that phrase is used, which we call usage scenarios). We further extend this representation with information extracted from external knowledge bases. The resulting representation incorporates both localized context and more general usage of the phrase and allows to detect its compositionality in a non-deterministic and contextual way. Empirical evaluation of our model on a dataset of phrase compositionality, manually collected by crowdsourcing contextual compositionality assessments, shows that our model outperforms state-of-the-art baselines notably on detecting phrase compositionality.

Computation and Language,Artificial Intelligence
Sparse Overcomplete Word Vector Representations

Manaal Faruqui,Yulia Tsvetkov,Dani Yogatama,Chris Dyer,Noah Smith

DOI: https://doi.org/10.48550/arXiv.1506.02004

2015-06-06

Abstract:Current distributed representations of words show little resemblance to theories of lexical semantics. The former are dense and uninterpretable, the latter largely based on familiar, discrete classes (e.g., supersenses) and relations (e.g., synonymy and hypernymy). We propose methods that transform word vectors into sparse (and optionally binary) vectors. The resulting representations are more similar to the interpretable features typically used in NLP, though they are discovered automatically from raw corpora. Because the vectors are highly sparse, they are computationally easy to work with. Most importantly, we find that they outperform the original vectors on benchmark tasks.

Computation and Language
Improving Distributed Representation Of Word Sense Via Wordnet Gloss Composition And Context Clustering

Chen Tao,Xu Ruifeng,He Yulan,Wang Xuan

DOI: https://doi.org/10.3115/v1/p15-2003

2015-01-01

Abstract:In recent years, there has been an increasing interest in learning a distributed representation of word sense. Traditional context clustering based models usually require careful tuning of model parameters, and typically perform worse on infrequent word senses. This paper presents a novel approach which addresses these limitations by first initializing the word sense embeddings through learning sentence-level embeddings from WordNet glosses using a convolutional neural networks. The initialized word sense embeddings are used by a context clustering based model to generate the distributed representations of word senses. Our learned representations outperform the publicly available embeddings on 2 out of 4 metrics in the word similarity task, and 6 out of 13 sub tasks in the analogical reasoning task.
Distributed representation of multi-sense words: A loss-driven approach

Saurav Manchanda,George Karypis

DOI: https://doi.org/10.1007/978-3-319-93037-4_27

2019-04-15

Abstract:Word2Vec's Skip Gram model is the current state-of-the-art approach for estimating the distributed representation of words. However, it assumes a single vector per word, which is not well-suited for representing words that have multiple senses. This work presents LDMI, a new model for estimating distributional representations of words. LDMI relies on the idea that, if a word carries multiple senses, then having a different representation for each of its senses should lead to a lower loss associated with predicting its co-occurring words, as opposed to the case when a single vector representation is used for all the senses. After identifying the multi-sense words, LDMI clusters the occurrences of these words to assign a sense to each occurrence. Experiments on the contextual word similarity task show that LDMI leads to better performance than competing approaches.

Computation and Language,Artificial Intelligence
Topic Modeling Using Distributed Word Embeddings

Ramandeep S Randhawa,Parag Jain,Gagan Madan

DOI: https://doi.org/10.48550/arXiv.1603.04747

2016-03-16

Abstract:We propose a new algorithm for topic modeling, Vec2Topic, that identifies the main topics in a corpus using semantic information captured via high-dimensional distributed word embeddings. Our technique is unsupervised and generates a list of topics ranked with respect to importance. We find that it works better than existing topic modeling techniques such as Latent Dirichlet Allocation for identifying key topics in user-generated content, such as emails, chats, etc., where topics are diffused across the corpus. We also find that Vec2Topic works equally well for non-user generated content, such as papers, reports, etc., and for small corpora such as a single-document.

Computation and Language
A Systematic Comparison of English Noun Compound Representations

Vered Shwartz

DOI: https://doi.org/10.48550/arXiv.1906.04772

2019-06-11

Computation and Language

Abstract:Building meaningful representations of noun compounds is not trivial since many of them scarcely appear in the corpus. To that end, composition functions approximate the distributional representation of a noun compound by combining its constituent distributional vectors. In the more general case, phrase embeddings have been trained by minimizing the distance between the vectors representing paraphrases. We compare various types of noun compound representations, including distributional, compositional, and paraphrase-based representations, through a series of tasks and analyses, and with an extensive number of underlying word embeddings. We find that indeed, in most cases, composition functions produce higher quality representations than distributional ones, and they improve with computational power. No single function performs best in all scenarios, suggesting that a joint training objective may produce improved representations.
Learning Distributed Representations of Sentences from Unlabelled Data

Felix Hill,Kyunghyun Cho,Anna Korhonen

DOI: https://doi.org/10.48550/arXiv.1602.03483

2016-02-10

Computation and Language

Abstract:Unsupervised methods for learning distributed representations of words are ubiquitous in today's NLP research, but far less is known about the best ways to learn distributed phrase or sentence representations from unlabelled data. This paper is a systematic comparison of models that learn such representations. We find that the optimal approach depends critically on the intended application. Deeper, more complex models are preferable for representations to be used in supervised systems, but shallow log-linear models work best for building representation spaces that can be decoded with simple spatial distance metrics. We also propose two new unsupervised representation-learning objectives designed to optimise the trade-off between training time, domain portability and performance.
Incorporating Linguistic Knowledge for Learning Distributed Word Representations.

Yan Wang,Zhiyuan Liu,Maosong Sun

DOI: https://doi.org/10.1371/journal.pone.0118437

IF: 3.7

2015-01-01

PLoS ONE

Abstract:Combined with neural language models, distributed word representations achieve significant advantages in computational linguistics and text mining. Most existing models estimate distributed word vectors from large-scale data in an unsupervised fashion, which, however, do not take rich linguistic knowledge into consideration. Linguistic knowledge can be represented as either link-based knowledge or preference-based knowledge, and we propose knowledge regularized word representation models (KRWR) to incorporate these prior knowledge for learning distributed word representations. Experiment results demonstrate that our estimated word representation achieves better performance in task of semantic relatedness ranking. This indicates that our methods can efficiently encode both prior knowledge from knowledge bases and statistical knowledge from large-scale text corpora into a unified word representation model, which will benefit many tasks in text mining.
Representation Learning of Multiword Expressions with Compositionality Constraint.

Minglei Li,Qin Lu,Yunfei Long

DOI: https://doi.org/10.1007/978-3-319-63558-3_43

2017-01-01

Abstract:Representations of multiword expressions (MWE) are currently learned either from context external to MWEs based on the distributional hypothesis or from the representations of component words based on some composition functions using the compositional hypothesis. However, a distributional method treats MWEs as a non-divisible unit without consideration of component words. Distributional methods also have the data sparseness problem, especially for MWEs. On the other hand, a compositional method can fail if a MWE is non-compositional. In this paper, we propose a hybrid method to learn the representation of MWEs from their external context and component words with a compositionality constraint. This method can make use of both the external context and component words. Instead of simply combining the two kinds of information, we use compositionality measure from lexical semantics to serve as the constraint. The main idea is to learn MWE representations based on a weighted linear combination of both external context and component words, where the weight is based on the compositionality of MWEs. Evaluation on three datasets shows that the performance of this hybrid method is more robust and can improve the representation.
A Unified Framework for Jointly Learning Distributed Representations of Word and Attributes.

Liqiang Niu,Xin-Yu Dai,Shujian Huang,Jiajun Chen

2015-01-01

Abstract:Distributed word representations have achieved great success in natural language processing (NLP) area. However, most distributed models focus on local context properties and learn task-specific representations individually, therefore lack the ability to fuse multi-attributes and learn jointly. In this paper, we propose a unified framework which jointly learns distributed representations of word and attributes: characteristics of word. In our models, we consider three types of attributes: topic, lemma and document. Besides learning distributed attribute representations, we find that using additional attributes is beneficial to improve word representations. Several experiments are conducted to evaluate the performance of the learned topic representations, document representations, and improved word representations, respectively. The experimental results show that our models achieve significant and competitive results.
Learning Bilingual Distributed Phrase Represenations for Statistical Machine Translation

Chaochao Wang,Deyi Xiong,Min Zhang,Chunyu Kit

2015-01-01

Abstract:Following the idea of using distributed semantic representations to facilitate the computation of semantic similarity between translation equivalents, we propose a novel framework to learn bilingual distributed phrase representations for machine translation. We first induce vector representations for words in the source and target language respectively, in their own semantic space. These word vectors are then used to create phrase representations via composition methods. In order to compute semantic similarity of phrase pairs in the same semantic space, we project phrase representations from the source-side semantic space onto the target-side semantic space via a neural network that is able to conduct nonlinear transformation between the two spaces. We integrate the learned bilingual distributed phrase representations into a hierarchical phrase-based translation system to validate the effectiveness of our proposed framework. Experiment results show that our method is able to significantly improve translation quality and outperform previous methods that only use word representations or linear semantic space transformation.
Unlocking the complexity of phrasal composition: An interplay between semantic features and linguistic relations

Shaonan Wang,Songhee Kim,Jeffrey R Binder,Liina Pylkkänen

DOI: https://doi.org/10.1016/j.cognition.2024.105986

IF: 4.011

Cognition

Abstract:Understanding the computational operations involved in conceptual composition is fundamental for theories of language. However, the existing literature on this topic remains fragmented, comprising disconnected theories from various fields. For instance, while formal semantic theories in Linguistics rely on type-driven interpretation without explicitly representing the conceptual content of lexical items, neurolinguistic research suggests that the brain is sensitive to conceptual factors during word composition. What is the relationship between these two types of theories? Do they describe two distinct aspects of composition, operating independently, or do they connect in some way during interpretation by our brain? To probe this, we explored how the mathematical operations explaining the combination of two words into a phrase are affected by the semantic content of items and the formal linguistic relations between the combining items. For six phrase types that varied properties relevant to type-driven interpretation such as modification vs. argument-saturation and modifier context sensitivity, we collected human ratings of experiential semantic features both for the phrases and for all the individual words within the phrases. We then compared the ability of different computational combination rules to explain the phrase ratings based on the individual word ratings. Our results indicate that composition operations are not one-size-fits-all but rather depend on both feature type and linguistic relation. For example, in intersective Adjective-Noun phrases, addition is used to merge attention-related features, while color features are predominantly determined by the first word's ratings. In the case of social features, the verb chiefly guides interpretation in Verb-Noun phrases, whereas in Noun-Noun phrases, the model employs multiplication to combine the social features of the nouns.
Learning to Compose Representations of Different Encoder Layers towards Improving Compositional Generalization

Lei Lin,Shuangtao Li,Yafang Zheng,Biao Fu,Shan Liu,Yidong Chen,Xiaodong Shi

2023-10-18

Abstract:Recent studies have shown that sequence-to-sequence (seq2seq) models struggle with compositional generalization (CG), i.e., the ability to systematically generalize to unseen compositions of seen components. There is mounting evidence that one of the reasons hindering CG is the representation of the encoder uppermost layer is entangled, i.e., the syntactic and semantic representations of sequences are entangled. However, we consider that the previously identified representation entanglement problem is not comprehensive enough. Additionally, we hypothesize that the source keys and values representations passing into different decoder layers are also entangled. Starting from this intuition, we propose \textsc{CompoSition} (\textbf{Compo}se \textbf{S}yntactic and Semant\textbf{i}c Representa\textbf{tion}s), an extension to seq2seq models which learns to compose representations of different encoder layers dynamically for different tasks, since recent studies reveal that the bottom layers of the Transformer encoder contain more syntactic information and the top ones contain more semantic information. Specifically, we introduce a \textit{composed layer} between the encoder and decoder to compose different encoder layers' representations to generate specific keys and values passing into different decoder layers. \textsc{CompoSition} achieves competitive results on two comprehensive and realistic benchmarks, which empirically demonstrates the effectiveness of our proposal. Codes are available at~\url{<a class="link-external link-https" href="https://github.com/thinkaboutzero/COMPOSITION" rel="external noopener nofollow">this https URL</a>}.

Computation and Language
A Structured Distributional Model of Sentence Meaning and Processing

Emmanuele Chersoni,Enrico Santus,Ludovica Pannitto,Alessandro Lenci,Philippe Blache,Chu-Ren Huang

DOI: https://doi.org/10.48550/arXiv.1906.07280

2019-06-18

Abstract:Most compositional distributional semantic models represent sentence meaning with a single vector. In this paper, we propose a Structured Distributional Model (SDM) that combines word embeddings with formal semantics and is based on the assumption that sentences represent events and situations. The semantic representation of a sentence is a formal structure derived from Discourse Representation Theory and containing distributional vectors. This structure is dynamically and incrementally built by integrating knowledge about events and their typical participants, as they are activated by lexical items. Event knowledge is modeled as a graph extracted from parsed corpora and encoding roles and relationships between participants that are represented as distributional vectors. SDM is grounded on extensive psycholinguistic research showing that generalized knowledge about events stored in semantic memory plays a key role in sentence comprehension. We evaluate SDM on two recently introduced compositionality datasets, and our results show that combining a simple compositional model with event knowledge constantly improves performances, even with different types of word embeddings.

Computation and Language
Compositional Mixture Representations for Vision and Text

Stephan Alaniz,Marco Federici,Zeynep Akata

DOI: https://doi.org/10.48550/arXiv.2206.06404

2022-06-13

Computer Vision and Pattern Recognition

Abstract:Learning a common representation space between vision and language allows deep networks to relate objects in the image to the corresponding semantic meaning. We present a model that learns a shared Gaussian mixture representation imposing the compositionality of the text onto the visual domain without having explicit location supervision. By combining the spatial transformer with a representation learning approach we learn to split images into separately encoded patches to associate visual and textual representations in an interpretable manner. On variations of MNIST and CIFAR10, our model is able to perform weakly supervised object detection and demonstrates its ability to extrapolate to unseen combination of objects.
Vocabulary for Universal Approximation: A Linguistic Perspective of Mapping Compositions

Yongqiang Cai

2024-05-23

Abstract:In recent years, deep learning-based sequence modelings, such as language models, have received much attention and success, which pushes researchers to explore the possibility of transforming non-sequential problems into a sequential form. Following this thought, deep neural networks can be represented as composite functions of a sequence of mappings, linear or nonlinear, where each composition can be viewed as a \emph{word}. However, the weights of linear mappings are undetermined and hence require an infinite number of words. In this article, we investigate the finite case and constructively prove the existence of a finite \emph{vocabulary} $V=\{\phi_i: \mathbb{R}^d \to \mathbb{R}^d | i=1,...,n\}$ with $n=O(d^2)$ for the universal approximation. That is, for any continuous mapping $f: \mathbb{R}^d \to \mathbb{R}^d$, compact domain $\Omega$ and $\varepsilon>0$, there is a sequence of mappings $\phi_{i_1}, ..., \phi_{i_m} \in V, m \in \mathbb{Z}_+$, such that the composition $\phi_{i_m} \circ ... \circ \phi_{i_1} $ approximates $f$ on $\Omega$ with an error less than $\varepsilon$. Our results demonstrate an unusual approximation power of mapping compositions and motivate a novel compositional model for regular languages.

Machine Learning,Dynamical Systems,Numerical Analysis

Distributed representations of words and phrases and their compositionality. arXiv 2013

A Model of Coherence Based on Distributed Sentence Representation.

A Gloss Composition and Context Clustering Based Distributed Word Sense Representation Model.

Semantic Processing Through Distributed Representation of Chinese Words

"The Sum of Its Parts": Joint Learning of Word and Phrase Representations with Autoencoders

Contextual Compositionality Detection with External Knowledge Bases andWord Embeddings

Sparse Overcomplete Word Vector Representations

Improving Distributed Representation Of Word Sense Via Wordnet Gloss Composition And Context Clustering

Distributed representation of multi-sense words: A loss-driven approach

Topic Modeling Using Distributed Word Embeddings

A Systematic Comparison of English Noun Compound Representations

Learning Distributed Representations of Sentences from Unlabelled Data

Incorporating Linguistic Knowledge for Learning Distributed Word Representations.

Representation Learning of Multiword Expressions with Compositionality Constraint.

A Unified Framework for Jointly Learning Distributed Representations of Word and Attributes.

Learning Bilingual Distributed Phrase Represenations for Statistical Machine Translation

Unlocking the complexity of phrasal composition: An interplay between semantic features and linguistic relations

Learning to Compose Representations of Different Encoder Layers towards Improving Compositional Generalization

A Structured Distributional Model of Sentence Meaning and Processing

Compositional Mixture Representations for Vision and Text

Vocabulary for Universal Approximation: A Linguistic Perspective of Mapping Compositions