Abstract:Wikidata is a collaborative multi-purpose Knowledge Graph (KG) with the unique feature of adding provenance data to the statements of items as a reference. More than 73% of Wikidata statements have provenance metadata; however, few studies exist on the referencing quality in this KG, focusing only on the relevancy and trustworthiness of external sources. While there are existing frameworks to assess the quality of Linked Data, and in some aspects their metrics investigate provenance, there are none focused on reference quality. We define a comprehensive referencing quality assessment framework based on Linked Data quality dimensions, such as completeness and understandability. We implement the objective metrics of the assessment framework as the Referencing Quality Scoring System – RQSS. The system provides quantified scores by which the referencing quality can be analyzed and compared. RQSS scripts can also be reused to monitor the referencing quality regularly. Due to the scale of Wikidata, we have used well-defined subsets to evaluate the quality of references in Wikidata using RQSS. We evaluate RQSS over three topical subsets: Gene Wiki, Music, and Ships, corresponding to three Wikidata WikiProjects, along with four random subsets of various sizes. The evaluation shows that RQSS is practical and provides valuable information, which can be used by Wikidata contributors and project holders to identify the quality gaps. Based on RQSS, the average referencing quality in Wikidata subsets is 0.58 out of 1. Random subsets (representative of Wikidata) have higher overall scores than topical subsets by 0.05, with Gene Wiki having the highest scores amongst topical subsets. Regarding referencing quality dimensions, all subsets have high scores in accuracy, availability, security, and understandability, but have weaker scores in completeness, verifiability, objectivity, and versatility. Although RQSS is developed based on the Wikidata RDF model, its referencing quality assessment framework can be applied to KGs in general.

Enabling Fine-grained RDF Data Completeness Assessment

Assessing Data Quality Within Available Context

Completeness Guarantees for Incomplete Ontology Reasoners: Theory and Practice

Enriching Wikidata with Linked Open Data

Completeness, Recall, and Negation in Open-world Knowledge Bases: A Survey

Non-Parametric Class Completeness Estimators for Collaborative Knowledge Graphs -- The Case of Wikidata

Commonsense Knowledge in Wikidata

Entrez Neuron RDFa: a pragmatic semantic web application for data integration in neuroscience research.

Browser-based Semantic Mapping Tool for Linked Data in Semantic Web.

Sapphire: Querying RDF Data Made Simple

Evaluating the Quality of RDF Data Sets on Common Vocabularies in the Social, Behavioral, and Economic Sciences

Querying RDF Data Using Dynamic Concise Bounded Description

A survey on SPARQL query relaxation under the lens of RDF reification

RDF 1.1: Knowledge Representation and Data Integration Language for the Web

Constraints to Validate RDF Data Quality on Common Vocabularies in the Social, Behavioral, and Economic Sciences

Ontology-based Data Access: A Study through Disjunctive Datalog, CSP, and MMSNP

Handling Wikidata Qualifiers in Reasoning

Mapping OMOP-CDM to RDF: Bringing Real-World-Data to the Semantic Web Realm

RQSS: Referencing quality scoring system for Wikidata

Reasoning over RDF Knowledge Bases using Deep Learning

In the Name of the Name: RDF literals, ER Attributes and the Potential to Rethink the Structures and Visualizations of Catalogs