Abstract:With the explosive growth of semantic data on the Web over the past years, many large-scale RDF knowledge bases with billions of facts are generating. This poses significant challenges for the storage and query of big RDF graphs. Current systems still have many limitations in processing big RDF graphs including scalability and real-time. In this paper, we introduce the SparkRDF, an elastic discreted RDF graph processing engine with distributed memory. To reduce the high I/O and communication cost in distributed processing platforms, SparkRDF implements SPARQL query based on Spark, a novel in-memory distributed computing framework for big data processing. All the intermediate results are modeled as Resilient Discreted SubGraph, which are cached in the distributed memory to support fast iterative join operations. To cut down the search space and avoid the overhead of memory, we split the RDF graph into the small Multi-layer Elastic SubGraph based on the relations and classes. For SPARQL query optimization, SparkRDF deploys a serials of optimization strategies, leading to effective reduction on the size of intermediate results, the number of joins and the cost of communication. Our extensive evaluation demonstrates that SparkRDF can efficiently implement non-selective joins faster than both current state-of-the-art distributed and centralized stores, while being able to process other queries in real time, scaling linearly to the amount of data.

A Survey of RDF Management Technologies and Benchmark Datasets

A Survey of Distributed RDF Data Management

Cloud Data Management: A Survey

Browsing RDF Data:State of Art Survey

Scalable RDF store based on HBase and MapReduce

Storage, Indexing, Query Processing, and Benchmarking in Centralized and Distributed RDF Engines: A Survey

A survey for managing temporal data in RDF

LITMUS: An Open Extensible Framework for Benchmarking RDF Data Management Solutions

A Survey of Big Data Research

When Things Matter: A Survey on Data-Centric Internet of Things

BANDAR: Benchmarking Snippet Generation Algorithms for (RDF) Dataset Search

A Survey on Mapping Semi-Structured Data and Graph Data to Relational Data

Benchmarking Graph Data Management and Processing Systems: A Survey

SparkRDF: Elastic Discreted RDF Graph Processing Engine with Distributed Memory

Data Storage and Access in Wireless Sensor Networks:A Survey

Graph-Based RDF Data Management

SparkRDF: In-Memory Distributed RDF Management Framework for Large-Scale Social Data.

Distributed Semantic Web Data Management in HBase and MySQL Cluster

Not All RDF is Created Equal: Investigating RDF Load Times on Resource-Constrained Devices

Benchmarking Big Data Systems: State-of-the-Art and Future Directions

When Things Matter: A Data-Centric View of the Internet of Things.