Abstract:The rise of diversified data engines has created the need for federated queries. A federated query can take a query and provide data analysis based on data from various data engines. Since the query data originates from multiple data engines, federated queries usually rely on join operation and data migration to complete the query and take a long time. The challenges of optimizing federated queries lie on join order selection and data migration coordination. However, enumerating all join orders is impractical because the set of join orders grows exponentially with the number of relations to be joined. To improve the performance of federated queries, we present a deep reinforcement learning-based approach on optimizing join order and join engine selection for federated queries and design an deep Q-network-based (DQN-based) optimizer. The DQN-based optimizer can generate join search policies that optimize the join order selection for datasets with a given cost model. Based on the DQN-based optimizer, we implement a federated query system Coral which can provide optimization for join order selection of federated queries. With the optimized join order, Coral can transform a federated query into a set of subqueries which will be assigned to and executed on different data engines. We also propose a subquery cache optimization to optimize data migration during the query execution. The extensive experimental evaluation demonstrates that Coral can significantly reduce the query latency of federated queries and achieve a speedup of up to 5.03 × compared to the cutting-edge federated query systems.

Multi-Query Optimization In Federated Rdf Systems

Optimizing Multi-Query Evaluation in Federated RDF Systems

A Cost-Driven Top-K Queries Optimization Approach on Federated RDF Systems

SPARQL Multi-Query Optimization.

FedTopK: Top-K Queries Optimization over Federated RDF Systems

A partition-based Summary-Graph-Driven Method for Efficient RDF Query Processing

Minimizing Communication Cost in Distributed Multi-Query Processing

Adaptive Multi-join Query Processing in PDBMS

New Distributed Spatial Query Optimization Approach by Using Query Analyzer

The Odyssey Approach for Optimizing Federated SPARQL Queries

Optimizing Federated Queries Based on the Physical Design of a Data Lake

Optimization of Data Retrievals in Processing Data Integration Queries

An empirical evaluation of cost-based federated SPARQL query processing engines

Energy-Efficient multi-query optimization over large-scale sensor networks

Coral: Federated Query Join Order Optimization Based on Deep Reinforcement Learning

SparkRDF: Elastic Discreted RDF Graph Processing Engine with Distributed Memory

FusionQuery: On-demand Fusion Queries over Multi-source Heterogeneous Data

Processing SPARQL Queries over Distributed RDF Graphs

Efficient Distributed Query Processing in Large RFID-enabled Supply Chains

Query Workload-based RDF Graph Fragmentation and Allocation

Optimization of sub-query processing in distributed data integration systems