Abstract:Semantic networks, such as the knowledge graph, can represent the knowledge leveraging the graph structure. Although the knowledge graph shows promising values in natural language processing, it suffers from incompleteness. This paper focuses on knowledge graph completion by predicting linkage between entities, which is a fundamental yet critical task. Semantic matching is a potential solution as it can deal with unseen entities, which the translational distance based methods struggle with. However, to achieve competitive performance as translational distance based methods, semantic matching based methods require large-scale datasets for the training purpose, which are typically unavailable in practical settings. Therefore, we employ the language model and introduce a novel knowledge graph architecture named LP-BERT, which contains two main stages: multi-task pre-training and knowledge graph fine-tuning. In the pre-training phase, three tasks are taken to drive the model to learn the relationship from triples by predicting either entities or relations. While in the fine-tuning phase, inspired by contrastive learning, we design a triple-style negative sampling in a batch, which greatly increases the proportion of negative sampling while keeping the training time almost unchanged. Furthermore, we propose a new data augmentation method utilizing the inverse relationship of triples to improve the performance and robustness of the model. To demonstrate the effectiveness of our method, we conduct extensive experiments on three widely-used datasets, WN18RR, FB15k-237, and UMLS. The experimental results demonstrate the superiority of our methods, and our approach achieves state-of-the-art results on WN18RR and FB15k-237 datasets. Significantly, Hits@10 indicator is improved by 5% from previous state-of-the-art result on the WN18RR dataset while reaching 100% on the UMLS dataset.

Enhancing Pre-Trained Language Representations Based on Contrastive Learning for Unsupervised Keyphrase Extraction

OssCSE: Overcoming Surface Structure Bias in Contrastive Learning for Unsupervised Sentence Embedding

Joint Keyphrase Chunking and Salience Ranking with BERT

AdaptiveUKE: Towards adaptive unsupervised keyphrase extraction with gated topic modeling

Attention-Seeker: Dynamic Self-Attention Scoring for Unsupervised Keyphrase Extraction

Simple Flow-Based Contrastive Learning for BERT Sentence Representations

Multi-level Contrastive Learning for Keyphrase Generation

Phrase-BERT: Improved Phrase Embeddings from BERT with an Application to Corpus Exploration

Capturing Global Informativeness in Open Domain Keyphrase Extraction

Pre-Trained Language Models for Keyphrase Prediction: A Review

Unsupervised Document Embedding via Contrastive Augmentation

Knowledgeable Salient Span Mask for Enhancing Language Models As Knowledge Base

Scientific Keyphrase Extraction: Extracting Candidates with Semi-supervised Data Augmentation.

Enhancing unsupervised keyphrase extraction through the integration of structural details in embedding-based approaches

Paying More Attention to Self-attention: Improving Pre-trained Language Models via Attention Guiding

Improving Weakly Supervised Phrase Grounding Via Visual Representation Contextualization with Contrastive Learning.

Contrastive Learning Models for Sentence Representations

Multi-task Pre-training Language Model for Semantic Network Completion

Enhancing Phrase Representation by Information Bottleneck Guided Text Diffusion Process for Keyphrase Extraction

C2BERT - Cross-contrast BERT for Chinese Biomedical Sentence Representation.

Contrastive Augmentation: An Unsupervised Learning Approach for Keyword Spotting in Speech Technology