Abstract:Semantic networks, such as the knowledge graph, can represent the knowledge leveraging the graph structure. Although the knowledge graph shows promising values in natural language processing, it suffers from incompleteness. This paper focuses on knowledge graph completion by predicting linkage between entities, which is a fundamental yet critical task. Semantic matching is a potential solution as it can deal with unseen entities, which the translational distance based methods struggle with. However, to achieve competitive performance as translational distance based methods, semantic matching based methods require large-scale datasets for the training purpose, which are typically unavailable in practical settings. Therefore, we employ the language model and introduce a novel knowledge graph architecture named LP-BERT, which contains two main stages: multi-task pre-training and knowledge graph fine-tuning. In the pre-training phase, three tasks are taken to drive the model to learn the relationship from triples by predicting either entities or relations. While in the fine-tuning phase, inspired by contrastive learning, we design a triple-style negative sampling in a batch, which greatly increases the proportion of negative sampling while keeping the training time almost unchanged. Furthermore, we propose a new data augmentation method utilizing the inverse relationship of triples to improve the performance and robustness of the model. To demonstrate the effectiveness of our method, we conduct extensive experiments on three widely-used datasets, WN18RR, FB15k-237, and UMLS. The experimental results demonstrate the superiority of our methods, and our approach achieves state-of-the-art results on WN18RR and FB15k-237 datasets. Significantly, Hits@10 indicator is improved by 5% from previous state-of-the-art result on the WN18RR dataset while reaching 100% on the UMLS dataset.

BAM! Born-Again Multi-Task Networks for Natural Language Understanding

MLKD-BERT: Multi-level Knowledge Distillation for Pre-trained Language Models

Conditionally Adaptive Multi-Task Learning: Improving Transfer Learning in NLP Using Fewer Parameters & Less Data

Multi-task Pre-training Language Model for Semantic Network Completion

One Teacher is Enough? Pre-trained Language Model Distillation from Multiple Teachers

Robustly Optimized and Distilled Training for Natural Language Understanding

Leveraging Large Language Models for Enhanced NLP Task Performance through Knowledge Distillation and Optimized Training Strategies

Multi-Grained Knowledge Distillation for Named Entity Recognition

Optimizing Multi-Task Learning for Enhanced Performance in Large Language Models

A Flexible Multi-Task Model for BERT Serving

Life-long Learning for Multilingual Neural Machine Translation with Knowledge Distillation

BERT and PALs: Projected Attention Layers for Efficient Adaptation in Multi-Task Learning

XtremeDistil: Multi-stage Distillation for Massive Multilingual Models

Self-Knowledge Distillation in Natural Language Processing

Task-Attentive Transformer Architecture for Continual Learning of Vision-and-Language Tasks Using Knowledge Distillation

BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding

Multitask Fine-Tuning and Generative Adversarial Learning for Improved Auxiliary Classification

Automatic Student Network Search for Knowledge Distillation.

LRC-BERT: Latent-representation Contrastive Knowledge Distillation for Natural Language Understanding

Improving task-agnostic BERT distillation with layer mapping search

Building a Multi-domain Neural Machine Translation Model using Knowledge Distillation