Abstract:Sentence embeddings encode natural language sentences as low-dimensional dense vectors. A great deal of effort has been put into using sentence embeddings to improve several important natural language processing tasks. Relation extraction is such an NLP task that aims at identifying structured relations defined in a knowledge base from unstructured text. A promising and more efficient approach would be to embed both the text and structured knowledge in low-dimensional spaces and discover semantic alignments or mappings between them. Although a number of techniques have been proposed in the literature for embedding both sentences and knowledge graphs, little is known about the structural and semantic properties of these embedding spaces in terms of relation extraction. In this paper, we investigate the aforementioned properties by evaluating the extent to which sentences carrying similar senses are embedded in close proximity sub-spaces, and if we can exploit that structure to align sentences to a knowledge graph. We propose a set of experiments using a widely-used large-scale data set for relation extraction and focusing on a set of key sentence embedding methods. We additionally provide the code for reproducing these experiments at <a class="link-external link-https" href="https://github.com/akalino/semantic-structural-sentences" rel="external noopener nofollow">this https URL</a>. These embedding methods cover a wide variety of techniques ranging from simple word embedding combination to transformer-based BERT-style model. Our experimental results show that different embedding spaces have different degrees of strength for the structural and semantic properties. These results provide useful information for developing embedding-based relation extraction methods.

A Brief Overview of Universal Sentence Representation Methods: A Linguistic View

Evaluation of sentence embeddings in downstream and linguistic probing tasks

Do Multi-Sense Embeddings Improve Natural Language Understanding?

Learning Universal Representations from Word to Sentence

Exploring Multilingual Syntactic Sentence Representations

Contextual Lensing of Universal Sentence Representations

BURT: BERT-inspired Universal Representation from Learning Meaningful Segment

A Comparative Study on Structural and Semantic Properties of Sentence Embeddings

A Comprehensive Survey of Sentence Representations: From the BERT Epoch to the ChatGPT Era and Beyond

Pre-training Universal Language Representation

Unsupervised Cross-Lingual Sentence Representation Learning via Linguistic Isomorphism

Universal Multimodal Representation for Language Understanding

A Comparative Study of Sentence Embedding Models for Assessing Semantic Variation

Sentence Similarity Computation in Question Answering Robot

Universal Text Representation from BERT: An Empirical Study

Simple Techniques for Enhancing Sentence Embeddings in Generative Language Models

An Unsupervised Sentence Embedding Method Bymutual Information Maximization.

Semantic Sentence Embeddings for Paraphrasing and Text Summarization

Probing Contextualized Sentence Representations with Visual Awareness

Sentence Representation Method Based on Multi-Layer Semantic Network

Representing Sentences as Low-Rank Subspaces