Abstract:Semantic networks, such as the knowledge graph, can represent the knowledge leveraging the graph structure. Although the knowledge graph shows promising values in natural language processing, it suffers from incompleteness. This paper focuses on knowledge graph completion by predicting linkage between entities, which is a fundamental yet critical task. Semantic matching is a potential solution as it can deal with unseen entities, which the translational distance based methods struggle with. However, to achieve competitive performance as translational distance based methods, semantic matching based methods require large-scale datasets for the training purpose, which are typically unavailable in practical settings. Therefore, we employ the language model and introduce a novel knowledge graph architecture named LP-BERT, which contains two main stages: multi-task pre-training and knowledge graph fine-tuning. In the pre-training phase, three tasks are taken to drive the model to learn the relationship from triples by predicting either entities or relations. While in the fine-tuning phase, inspired by contrastive learning, we design a triple-style negative sampling in a batch, which greatly increases the proportion of negative sampling while keeping the training time almost unchanged. Furthermore, we propose a new data augmentation method utilizing the inverse relationship of triples to improve the performance and robustness of the model. To demonstrate the effectiveness of our method, we conduct extensive experiments on three widely-used datasets, WN18RR, FB15k-237, and UMLS. The experimental results demonstrate the superiority of our methods, and our approach achieves state-of-the-art results on WN18RR and FB15k-237 datasets. Significantly, Hits@10 indicator is improved by 5% from previous state-of-the-art result on the WN18RR dataset while reaching 100% on the UMLS dataset.

SJTU-NICT at MRP 2019 - Multi-Task Learning for End-to-End Uniform Semantic Graph Parsing.

SJTU at MRP 2019 - A Transition-Based Multi-Task Parser for Cross-Framework Meaning Representation Parsing.

Multilingual Dependency-based Syntactic and Semantic Parsing.

ShanghaiTech at MRP 2019: Sequence-to-Graph Transduction with Second-Order Edge Inference for Cross-Framework Meaning Representation Parsing

HIT-SCIR at MRP 2019: A Unified Pipeline for Meaning Representation Parsing Via Efficient Training and Effective Encoding.

A Syntax-aware Multi-task Learning Framework for Chinese Semantic Role Labeling

NJU-Parser: Achievements on Semantic Dependency Parsing.

HLT@SUDA at SemEval 2019 Task 1: UCCA Graph Parsing as Constituent Tree Parsing.

A Sketch-Based System for Semantic Parsing

A Joint Syntactic and Semantic Dependency Parsing System based on Maximum Entropy Models.

A Cascaded Syntactic and Semantic Dependency Parsing System.

HIT-SCIR at MRP 2020: Transition-based Parser and Iterative Inference Parser.

A Simple Yet Effective Joint Training Method for Cross-Lingual Universal Dependency Parsing.

An End-to-End Scalable Iterative Sequence Tagging with Multi-Task Learning.

SUDA-Alibaba at MRP 2019: Graph-Based Models with BERT.

Exploiting Multi-typed Treebanks for Parsing with Deep Multi-task Learning.

A Hierarchical Multi-Task Learning Framework for Semantic Annotation in Tabular Data

Multi-task Pre-training Language Model for Semantic Network Completion

A Fast and Lightweight System for Multilingual Dependency Parsing.

An Iterative Approach for Joint Dependency Parsing and Semantic Role Labeling.