Abstract:With the explosive growth of semantic data on the Web over the past years, many large-scale RDF knowledge bases with billions of facts are generating. This poses significant challenges for the storage and query of big RDF graphs. Current systems still have many limitations in processing big RDF graphs including scalability and real-time. In this paper, we introduce the SparkRDF, an elastic discreted RDF graph processing engine with distributed memory. To reduce the high I/O and communication cost in distributed processing platforms, SparkRDF implements SPARQL query based on Spark, a novel in-memory distributed computing framework for big data processing. All the intermediate results are modeled as Resilient Discreted SubGraph, which are cached in the distributed memory to support fast iterative join operations. To cut down the search space and avoid the overhead of memory, we split the RDF graph into the small Multi-layer Elastic SubGraph based on the relations and classes. For SPARQL query optimization, SparkRDF deploys a serials of optimization strategies, leading to effective reduction on the size of intermediate results, the number of joins and the cost of communication. Our extensive evaluation demonstrates that SparkRDF can efficiently implement non-selective joins faster than both current state-of-the-art distributed and centralized stores, while being able to process other queries in real time, scaling linearly to the amount of data.

Scalable Online Analysis of Semantic Web Data

HadoopRDF: a scalable semantic data analytical engine

SparkRDF: Elastic Discreted RDF Graph Processing Engine with Distributed Memory

Effective and efficient semantic web data management over DB2.

HadoopRDF : A Scalable RDF Data Analysis System

Fast Processing SPARQL Queries on Large RDF Data

An Approach to RDF(S) Query, Manipulation and Inference on Databases

Towards Efficient Distributed SPARQL Queries on Linked Data.

Processing SPARQL Queries over Linked Data-A Distributed Graph-based Approach.

SparkRDF: In-Memory Distributed RDF Management Framework for Large-Scale Social Data.

Scalable RDF store based on HBase and MapReduce

Scalable SAPRQL querying processing on large RDF data in cloud computing environment

Scalable SPARQL Querying Using Path Partitioning

Efficient query evaluation techniques over large amount of distributed linked data

S-store: An Engine for Large RDF Graph Integrating Spatial Information.

Scalable RDF Graph Querying Using Cloud Computing

Processing SPARQL Queries over Distributed RDF Graphs

Storage, Indexing, Query Processing, and Benchmarking in Centralized and Distributed RDF Engines: A Survey

Implementation and Optimization of RDF Query Using Hadoop.

An Improved Distributed Query for Large-Scale RDF Data

RDF partitioning for scalable SPARQL query processing