KACC: A Multi-task Benchmark for Knowledge Abstraction, Concretization and Completion.

Jie Zhou,Shengding Hu,Xin Lv,Cheng Yang,Zhiyuan Liu,Wei Xu,Jie Jiang,Juanzi Li,Maosong Sun
DOI: https://doi.org/10.18653/v1/2021.findings-acl.153
2020-01-01
Findings
Abstract:A comprehensive knowledge graph (KG) contains an instance-level entity graphand an ontology-level concept graph. The two-view KG provides a testbed formodels to "simulate" human's abilities on knowledge abstraction,concretization, and completion (KACC), which are crucial for human to recognizethe world and manage learned knowledge. Existing studies mainly focus onpartial aspects of KACC. In order to promote thorough analyses for KACCabilities of models, we propose a unified KG benchmark by improving existingbenchmarks in terms of dataset scale, task coverage, and difficulty.Specifically, we collect new datasets that contain larger concept graphs,abundant cross-view links as well as dense entity graphs. Based on thedatasets, we propose novel tasks such as multi-hop knowledge abstraction (MKA),multi-hop knowledge concretization (MKC) and then design a comprehensivebenchmark. For MKA and MKC tasks, we further annotate multi-hop hierarchicaltriples as harder samples. The experimental results of existing methodsdemonstrate the challenges of our benchmark. The resource is available athttps://github.com/thunlp/KACC.
What problem does this paper attempt to address?