Abstract:The use of large-scale machine learning methods is becoming ubiquitous in many applications ranging from business intelligence to self-driving cars. These methods require a complex computation pipeline consisting of various types of operations, e.g., relational operations for pre-processing or post-processing the dataset, and matrix operations for core model computations. Many existing systems focus on efficiently processing matrix-only operations, and assume that the inputs to the relational operators are already pre-computed and are materialized as intermediate matrices. However, the input to a relational operator may be complex in machine learning pipelines, and may involve various combinations of matrix operators. Hence, it is critical to realize scalable and efficient relational query processors that directly operate on big matrix data. This paper presents new efficient and scalable relational query processing techniques on big matrix data for in-memory distributed clusters. The proposed techniques leverage algebraic transformation rules to rewrite query execution plans into ones with lower computation costs. A distributed query plan optimizer exploits the sparsity-inducing property of merge functions as well as Bloom join strategies for efficiently evaluating various flavors of the join operation. Furthermore, optimized partitioning schemes for the input matrices are developed to facilitate the performance of join operations based on a cost model that minimizes the communication overhead.The proposed relational query processing techniques are prototyped in Apache Spark. Experiments on both real and synthetic data demonstrate that the proposed techniques achieve up to two orders of magnitude performance improvement over state-of-the-art systems on a wide range of applications.

Multi-Relational Algebra and Its Applications to Data Insights

A Multi-Dimensional Data Model and its Application

Multi-SQL: an Automatic Multi-model Data Management System.

Generating Multidimensional Schemata from Relational Aggregation Queries

Auto-Tables: Synthesizing Multi-Step Transformations to Relationalize Tables without Using Examples

Exposing Multi-Relational Networks to Single-Relational Network Analysis Algorithms

Multi-SQL: An extensible multi-model data query language

An Algebra for Online Analytical Processing Data Cube

An XML-based schema translation method for relational data sharing and exchanging

Research on Multi-relational Data Mining Approaches

Scalable Relational Query Processing on Big Matrix Data

An Algebra for Visual Analysis

AQUA: Automatic Collaborative Query Processing in Analytical Database

Multi-dimensional Analysis of Industrial Big Data Based JSON Document

A General Multi-relational Classification Approach Using Feature Generation and Selection.

A Survey of Multi-relational Frequent Pattern Discovery

Auto-Differentiation of Relational Computations for Very Large Scale Machine Learning

Multi-source relational data fusion

Deriving Application Level Relationships by Analysing the Run Time Behaviour of Simple and Complex SQL Queries

RASQL: A Powerful Language and Its System for Big Data Applications.

Graph Processing in RDBMSs.